253 commits

Author	SHA1	Message	Date
NotoriousRebel	b994a5d2f4	Added filtering method in constants.py and applied it to hostnames and emails in theHarvester.py	2018-12-23 19:54:56 -05:00
NotoriousRebel	f8ba4ebb2c	Added new line when printing users from Linkedin, Twitter, googleplus, and google-profiles.	2018-12-23 19:25:34 -05:00
NotoriousRebel	485d17c984	Wrapped pgp search inside try-except statement.	2018-12-23 17:27:30 -05:00
NotoriousRebel	b2d39a4d4b	Added securityTrails to invalid message in theHarvester.py	2018-12-23 17:05:12 -05:00
NotoriousRebel	c4bb668eca	Readding changes that were made with updated reporting from master.	2018-12-23 17:02:34 -05:00
NotoriousRebel	3d9fbf4411	Syncing.	2018-12-23 17:00:03 -05:00
NotoriousRebel	3f4f24eb97	Wrapped baidu search in try-except	2018-12-23 16:54:31 -05:00
NotoriousRebel	b895a5334f	Sorted emails and made them unique.	2018-12-23 16:12:27 -05:00
NotoriousRebel	ad21a4999d	Fixed cosmetic issues.	2018-12-23 15:32:30 -05:00
NotoriousRebel	09257e2247	Added class to handle missing api key and CSE id inside constants.py, implemented in respectful classes and inside theHarvester when it catches an exception it continues the program instead of sys.exit()	2018-12-23 15:23:01 -05:00
jzold	783ea640ca	Merge pull request #27 from jzold/report ... sync master with report	2018-12-23 17:21:34 +00:00
Janos Zold	831d1f6e1b	fix Censys plugin and filter out non-relevant domains yielded by the certificates search results	2018-12-23 13:13:08 +00:00
jzold	7ca0df7741	Merge branch 'censys' into master	2018-12-23 08:06:45 +00:00
NotoriousRebel	2394ef1646	Cleaned up cosmetic issues.	2018-12-22 23:33:35 -05:00
Christian Martorella	d7f595516c	Merge branch 'master' into master	2018-12-22 22:52:08 +01:00
Christian Martorella	4fc780fe19	Fixed and expanded Censys ... Expanded Censys to search certificates,hosts, ips.	2018-12-22 22:49:59 +01:00
NotoriousRebel	70188d0d75	Added security trails api back in theHarvester and fixed trello.	2018-12-22 15:48:46 -05:00
NotoriousRebel	8dbbb4d82a	Removed meta from url in trello and synced with master.	2018-12-22 15:43:53 -05:00
Christian Martorella	53703ffd0f	Minor Lee fixes	2018-12-22 21:29:11 +01:00
NotoriousRebel	0c8d1fc4ab	Fixed cosmetic things.	2018-12-22 13:46:27 -05:00
NotoriousRebel	6011dadbd7	Cleaned up trello search and modified theHarvester to utilize Trello to return hostnames and Trello urls.	2018-12-22 13:04:00 -05:00
Janos Zold	4b8f994999	[FIX]: printing host/domain name results to console	2018-12-22 17:56:44 +00:00
NotoriousRebel	2a5d3221f1	Syncingþ ... Merge remote-tracking branch 'upstream/master'	2018-12-21 21:16:31 -05:00
NotoriousRebel	b959971423	Fixed trello plugin to properly extract urls, reflected in theHarvester.py	2018-12-21 20:14:30 -05:00
NotoriousRebel	52b7a7d427	Integrated security trails into all mode, and touched up minor cosmetics.	2018-12-21 17:47:15 -05:00
NotoriousRebel	d76b3b8daa	Added Security Trails API as plugin and added to theHarvester, also created parser for it.	2018-12-21 15:17:51 -05:00
Janos Zold	3debf86aa8	censys fix: add unique items to the db and fixing censys with -all switch	2018-12-20 20:50:48 +00:00
Janos Zold	f52af72e33	merge upstream and resolve conflicts	2018-12-20 20:19:32 +00:00
Janos Zold	498b14affa	censys plugin update: harvest hosts/domain names from certificates search	2018-12-20 19:41:59 +00:00
Laramies	8b88a662df	Update theHarvester.py ... Fixed GoogleCSE, Added UA randomization, and minor fixes.	2018-12-20 00:01:07 +01:00
NotoriousRebel	8beeae7e0b	resyncing.	2018-12-19 14:39:33 -05:00
NotoriousRebel	e34263cd55	Resyncing	2018-12-19 14:34:48 -05:00
jzold	27753206a7	Merge pull request #20 from jzold/master ... sync with master	2018-12-19 15:52:19 +00:00
jzold	fd55d883c8	Merge pull request #19 from jzold/master ... bugfix: censys plugin - redundant results saved in the DB	2018-12-19 15:51:36 +00:00
jzold	aa017a4e68	bugfix: censys plugin - redundant results saved in the DB	2018-12-19 15:49:25 +00:00
jzold	b57710cb82	Merge pull request #18 from jzold/master ... sync with master	2018-12-19 15:30:16 +00:00
jzold	a568484a92	Merge pull request #17 from jzold/master ... sync with master	2018-12-19 15:27:47 +00:00
jzold	343a42355d	bugfix: avoid duplicate entries saved in the DB using the all_* variables	2018-12-19 15:12:32 +00:00
Janos Zold	a24aa3e862	New reporting module format with contents	2018-12-19 09:09:01 +00:00
Janos Zold	eabbb30c4d	New reporting module with new format and contents	2018-12-19 09:08:42 +00:00
jzold	06290434f0	Fix plugins saving email results as type "host" in the DB	2018-12-18 22:53:28 +00:00
NotoriousRebel	239b0f31f9	Implemented randomization within plugins and cleaned up small bugs.	2018-12-18 00:21:05 -05:00
Laramies	61f3637967	Minor fixes	2018-12-18 00:14:42 +01:00
Christian Martorella	2c0eb3108b	Merge pull request #119 from jzold/report ... Update new report module	2018-12-17 22:03:09 +01:00
jzold	ffdfd6cc75	added theHarvester scan statistics table to the report	2018-12-17 20:36:30 +00:00
Thomas Defise	24ed98e373	Change hunterio to hunter	2018-12-17 17:05:11 +01:00
Thomas Defise	96554418e3	test	2018-12-17 14:29:48 +01:00
jzold	303f4465c1	Merge pull request #9 from laramies/master ... sync with upstream master	2018-12-16 18:50:48 +00:00
jzold	1460d61299	Merge branch 'report' of https://github.com/jzold/theHarvester into report	2018-12-16 16:37:47 +00:00
jzold	c966acf537	added scanning history report for domain	2018-12-16 16:37:36 +00:00
Laramies	c2359c0e71	Minor fix	2018-12-16 16:22:30 +01:00
Christian Martorella	17e6825dbe	Merge branch 'master' into report	2018-12-16 15:59:07 +01:00
NotoriousRebel	2423811e2e	Cleaned up illogical whitespaces.	2018-12-15 22:33:11 -05:00
NotoriousRebel	6b4b0e5ee3	Resyncing and fixed program crashing if using search engine that doesn't define all_ip, and made print statements more logical.	2018-12-15 22:28:52 -05:00
NotoriousRebel	1d2a8401d1	Fixing tool being broken if using search engine where all_ip is not defined.	2018-12-15 22:15:41 -05:00
NotoriousRebel	f3597ee267	Syncing.	2018-12-15 22:07:37 -05:00
NotoriousRebel	b1d979d335	Synching with master.	2018-12-15 22:01:39 -05:00
jzold	296ce7282b	Added new reporting module	2018-12-15 17:22:02 +00:00
jzold	d4004d6844	standardizng db for reporting, adding email as a resource type and saving resolved IP addresses in the db	2018-12-14 14:39:16 +00:00
Christian Martorella	7f1addee25	Merge pull request #113 from jzold/cymon ... Added Cymon search engine to theHarvester	2018-12-14 10:03:33 +01:00
Laramies	681ed17233	Vesion update	2018-12-11 22:10:08 +01:00
Laramies	e533a964dc	Merge branch 'master' of https://github.com/laramies/theHarvester	2018-12-11 22:07:57 +01:00
Laramies	887776e37a	Minor fixes ... XML save order	2018-12-11 22:07:48 +01:00
Christian Martorella	bd327a78d1	Merge pull request #114 from jzold/master ... Fixing typo in theHarvester help	2018-12-10 23:06:15 +01:00
Christian Martorella	978d0e7cf3	Merge pull request #112 from jzold/censys ... Censys page search fix	2018-12-10 23:05:54 +01:00
jzold	aef5a53fa8	Fixing typo in theHarvester help	2018-12-10 18:41:25 +00:00
jzold	703111b660	printing cymon search results	2018-12-09 18:45:50 +00:00
jzold	fef88abc1d	adding cymon	2018-12-09 17:51:48 +00:00
jzold	4cfd5b65cc	added "censys" to theHarvester usage help	2018-12-06 17:47:14 +00:00
jzold	505a71781d	fix: optimize search with minimum number of page queries	2018-12-06 16:32:25 +00:00
H. Kamran	3f1b9d7b13	Replace "Request" with "Requests", the actual module name	2018-12-01 19:38:08 -08:00
Laramies	e95a1c6f1a	Merge branch 'master' of https://github.com/laramies/theHarvester	2018-12-01 10:41:51 +01:00
Laramies	e1ef6288ba	Added Trello Search ... New plugin to search public Trello boards	2018-12-01 10:39:01 +01:00
jzold	e7ef2c142c	fix for adding only unique IP and hostnames to SQLite	2018-11-30 21:20:20 +00:00
jzold	a0bf7044ca	censys fixing pagination	2018-11-30 20:57:12 +00:00
NotoriousRebel	8953b4d100	Made threatcrowd parse totalresults not just results, got censysparser to get pages and crawl them.	2018-11-29 16:48:36 -05:00
Laramies	bd67719f49	Merge branch 'master' of https://github.com/laramies/theHarvester	2018-11-29 22:28:40 +01:00
Laramies	632632c7a8	Update theHarvester.py	2018-11-29 22:28:37 +01:00
Christian Martorella	eae4fe5927	Merge pull request #107 from NotoriousRebel/master ... Modified discovery/crtsh.py to properly do searches.	2018-11-29 22:24:51 +01:00
NotoriousRebel	48cc8ce0de	Updating ... Merge remote-tracking branch 'upstream/master'	2018-11-28 17:16:44 -05:00
Laramies	05feda73e7	Version update	2018-11-28 21:34:26 +01:00
NotoriousRebel	f3788c6b41	Added method to get crt.sh ids and crawl them.	2018-11-27 22:05:51 -05:00
Laramies	590489fd5c	Fixed GoogleCSE ... Fixed API message, and error while executing due to order of storing data.	2018-11-27 23:29:05 +01:00
NotoriousRebel	20f22b57bd	fixed local variable error being referenced before defined in googleCSE search, also removed extra whitespace.	2018-11-26 14:07:20 -05:00
Laramies	36ceaab174	Google Certificates ... Added new search engine Google Certificate Transparency and fixed temporarily Shodan	2018-11-23 00:31:21 +01:00
Laramies	f53308fe00	Google-CRT changes ... Google-CRT changes	2018-11-22 22:51:31 +01:00
Christian Martorella	92b1a87246	Merge branch 'master' into master	2018-11-22 22:20:06 +01:00
NotoriousRebel	d21e2846d7	Cleaned up imports and added proper casting from bytes to str.	2018-11-20 22:17:41 -05:00
Matt	986112a207	Update theHarvester.py ... Removed debug print statements.	2018-11-20 13:01:26 -05:00
NotoriousRebel	157ff0dc69	Converted more code to python3 shodan works for the most part.	2018-11-20 12:04:57 -05:00
NotoriousRebel	4ba7c685c0	CleaConverted more of the code to python3 and verified that running code with all search engines works except crtsh gets inside loop I think.	2018-11-19 21:27:53 -05:00
NotoriousRebel	d38b1c4473	Replaced r.content with r.text in googlesearch to work properly.	2018-11-19 15:57:42 -05:00
NotoriousRebel	35bbfeaf44	Replacing httplib with requests.	2018-11-19 15:28:38 -05:00
NotoriousRebel	554ab59176	Cleaned up more python2 code and had to fix implicit conversion not happening in python3.	2018-11-19 15:00:19 -05:00
NotoriousRebel	d5de70b1c6	Working on small semantic error.	2018-11-15 13:03:45 -05:00
NotoriousRebel	2a20d3ff47	Starting the change from python2 to python3.7	2018-11-14 22:13:52 -05:00
jzold	9003c54437	Fix issue #71: -f now exports both in HTML and XML	2018-11-11 14:35:47 +00:00
jzold	8c41554f1e	sync with origin master	2018-11-11 14:24:58 +00:00
NotoriousRebel	b92cdf07f5	Modified hunter search engine so user just specificies key in huntersearch.py!	2018-11-06 17:12:20 -05:00
jzold	de53ecba96	Update theHarvester with BS4 module check	2018-11-06 07:45:28 +00:00
NotoriousRebel	15b9ff0192	Added try catch around threatcrowd incase it fails, so it doesn't crash program.	2018-11-05 18:58:15 -05:00
NotoriousRebel	36eacf2267	Reworking hunter search engine to integrate more logically and better.	2018-11-05 18:54:32 -05:00
NotoriousRebel	444e7e1d96	Added hunter search engine and made sure it doesn't overwite previous data.	2018-11-05 15:15:36 -05:00
NotoriousRebel	dc8e555c15	Updated huntersearch to work properly, have to verify it doesn't overwrite existing data.	2018-11-05 14:54:59 -05:00
NotoriousRebel	bcdb9a66d4	Reworking huntersearch to integrate better and more logically.	2018-11-04 19:19:48 -05:00
NotoriousRebel	73642b5843	Reworking hunter search engine into theHarvester.py to work properly.	2018-11-04 19:15:07 -05:00
NotoriousRebel	e7b22b94b7	Added hunter search engine and just need to test it, checked to make sure user key was not empty.	2018-11-04 17:26:34 -05:00
jzold	6cf3d18902	Censys implementation	2018-11-02 23:04:20 +00:00
NotoriousRebel	9d3c2b79db	Added huntersearch class for huntersearches, requires an api key.	2018-11-02 09:36:51 -04:00
NotoriousRebel	e6d96d4ffe	Added print statement to explain what -g flag does.	2018-10-26 16:32:31 -04:00
NotoriousRebel	6e55a63c40	Made boolean to be an arguement instead of necessary in class as only used in one spot which is inside function.	2018-10-26 16:15:43 -04:00
NotoriousRebel	ebf4961363	cleaned up code to resolve conflict hopefully.	2018-10-26 15:42:47 -04:00
NotoriousRebel	befb9e502a	Removed code that was leading to program not working properly.	2018-10-25 19:55:52 -04:00
NotoriousRebel	03c2e35457	Readded code I accidently deleted.	2018-10-24 21:16:29 -04:00
NotoriousRebel	9cdfdc1ef4	Made one small change to usage.	2018-10-24 20:51:13 -04:00
NotoriousRebel	108ab7d5cf	Shrunk dorks.txt to 200 to decrease chances of getting caught by google.	2018-10-24 20:49:48 -04:00
NotoriousRebel	c1fcf35aef	Reworked logic, increased dorks.txt, and need to do a few test runs to verifiy dorking increases output.	2018-10-24 15:09:23 -04:00
NotoriousRebel	568ace4419	Removed google dork class, added more dorks, and added boolean to indicate if user wants google dorks in google search.	2018-10-23 16:54:59 -04:00
NotoriousRebel	c2f86f7c54	Modified google dork to correctly generate urls and added user agent list. Also had to import it for some reason, may be an IDE issue, have to test later to confirm.	2018-10-19 16:23:49 -04:00
NotoriousRebel	b964623a6b	Started working on google dorking, created new file: googledork.py	2018-10-18 13:14:30 -04:00
laramies	973220d3d1	Release 3.0 ... Added Port scanning, Subdomain takeover, Fixed Shodan, improved Dns bruteforce and added SQLite support to store the results.	2018-08-08 21:34:10 +02:00
laramies	3d526edcb8	Merge branch 'master' of https://github.com/laramies/theHarvester	2018-04-16 13:55:56 +02:00
laramies	f971f6acfe	Dns brute force fixes	2018-04-16 13:55:52 +02:00
Christian Martorella	95e32aa343	3.0 ... Added port_scanning and domain Takeover checks	2018-04-16 09:32:10 +02:00
Christian Martorella	12921a65c6	Local db, output colors, threatcrowd	2018-03-22 23:32:50 +01:00
Christian Martorella	25553762d2	Minor fix ... Added a results header to new engines.	2017-09-23 00:35:49 +02:00
Christian Martorella	20d62adfb5	Adding dis-names.txt ... The file for dns names brute forcing was missing and people needed to add it manually.	2017-09-23 00:27:49 +02:00
Christian Martorella	d0e29f6122	New engines ... Added Virustotal, CRT.sh and Netcraft. More results :)	2017-09-22 23:05:00 +02:00
laramies	eb08d32ecc	Added missing search engines. ... Added missing search engines, fixed search engine handling.	2016-10-11 10:11:14 +01:00
laramies	0f9a670c44	2.7	2016-04-19 00:47:24 +01:00
laramies	e14dafdb22	2.7 Release ... Changelog for 2.7 and Banner version	2016-04-19 00:37:25 +01:00
laramies	07988b6828	Merge remote-tracking branch 'origin/master' ... # Conflicts: # theHarvester.py	2016-04-19 00:15:09 +01:00
laramies	9998bcc100	Merge pull request #37 from surfuga/xml-output ... xml-output with shodan and more	2016-04-19 00:07:25 +01:00
laramies	12abae2b44	Email parser ... Changed Regexp for Email parsing, supposed to produce cleaner results.	2016-04-19 00:00:45 +01:00
Brehret Nish	fba0528912	fix incorrect file closing	2016-03-05 10:25:44 -05:00
Pau Rodriguez	08fd9d7eba	xml-output: host,vhost added subfield ip and hostname, adding shodan output ... adding sorted(set()) to all_emails, all_hosts, vhosts; using full in the xml-output	2016-02-13 15:28:46 +01:00
Robin Wood	d095d6c570	Fixed the order of two lines in the command line example and ... a couple of PEP 8 complaints	2015-12-02 16:32:19 +00:00
laramies	a5e76ea790	Merge pull request #25 from duckspeaker/remove-123people ... remove now defunct 123people	2015-07-05 17:11:49 +01:00
laramies	65a3edff17	Merge pull request #19 from initbar/master ... Fix for Issue #9.	2015-07-05 17:07:15 +01:00
laramies	c6d21905f9	Change request import check at the beginning of the main file, to avoid ... the error 23.	2015-07-05 17:03:13 +01:00
Joe Gallo	338578c799	remove 123people	2015-07-05 04:31:39 -04:00
Bret N	8121b66b33	Set a temporary tag parser.	2015-05-27 21:30:52 -04:00
laramies	9b16a8a810	Merge branch 'pr/11'	2015-05-10 23:33:10 +01:00
laramies	4e49ea3c30	2.6 ... Requests check.	2015-05-10 23:29:44 +01:00
laramies	1a38153714	2.5a ... Updated the readme with changes and UI.	2015-05-10 23:21:09 +01:00
laramies	542ac20012	Merge pull request #15 from initbar/master ... fixed minor UI	2015-05-10 23:10:59 +01:00
BN	9c28e7de05	fixed minor UI	2015-03-28 15:14:26 -04:00
Adam Compton	9a6dce8e5b	added new search engines	2015-03-25 12:55:15 -04:00
Cameron Norman	e038472ed4	Improve usage() when installed in /usr/bin	2015-01-15 17:02:11 -08:00
dcell	537aafb348	UI help corrections	2015-01-03 04:17:47 +00:00