theHarvester/README.md

![theHarvester](https://github.com/laramies/theHarvester/blob/master/theHarvester-logo.png)

[![Build Status](https://travis-ci.com/laramies/theHarvester.svg?branch=master)](https://travis-ci.com/laramies/theHarvester) [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/laramies/theHarvester.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/laramies/theHarvester/context:python)
[![Rawsec's CyberSecurity Inventory](https://inventory.rawsec.ml/img/badges/Rawsec-inventoried-FF5050_flat_without_logo.svg)](https://inventory.rawsec.ml/)

What is this?
-------------
theHarvester is a very simple to use, yet powerful and effective tool designed to be used in the early statges of a<br>
penetration test or red team engagement. Use it for open source intelligence (OSINT) gathering to help determine a<br>
company's external threat landscape on the internet. The tool gathers emails, names, subdomains, IPs and URLs using<br>
multiple public data sources that include:

Passive:
--------
* baidu: Baidu search engine - www.baidu.com

* bing: Microsoft search engine - www.bing.com

* bingapi: Microsoft search engine, through the API (Requires an API key, see below.)

* Bufferoverun: Uses data from Rapid7's Project Sonar - www.rapid7.com/research/project-sonar/

* CertSpotter: Cert Spotter monitors Certificate Transparency logs - https://sslmate.com/certspotter/

* crtsh: Comodo Certificate search - https://crt.sh/

* dnsdumpster: DNSdumpster search engine - https://dnsdumpster.com/

* dogpile: Dogpile search engine - www.dogpile.com

* duckduckgo: DuckDuckGo search engine - www.duckduckgo.com

* Exalead: a Meta search engine - www.exalead.com/search

* github-code: GitHub code search engine (Requires a GitHub Personal Access Token, see below.) - www.github.com

* google: Google search engine (Optional Google dorking.) - www.google.com

* hunter: Hunter search engine (Requires an API key, see below.) - www.hunter.io

* intelx: Intelx search engine (Requires an API key, see below.) - www.intelx.io

* linkedin: Google search engine, specific search for LinkedIn users - www.linkedin.com

* netcraft: Internet Security and Data Mining - www.netcraft.com

* otx: AlienVault Open Threat Exchange - otx.alienvault.com

* securityTrails: Security Trails search engine, the world's largest repository of historical DNS data<br>
  (Requires an API key, see below.) - www.securitytrails.com

* shodan: Shodan search engine, will search for ports and banners from discovered hosts - www.shodanhq.com

* Spyse: Web research tools for professionals (Requires an API key.) - spyse.com

* Suip: Web research tools that can take over 10 minutes to run, but worth the wait - suip.biz

* threatcrowd: Open source threat intelligence - www.threatcrowd.org

* trello: Search trello boards (Uses Google search.)

* twitter: Twitter accounts related to a specific domain (Uses Google search.)

* vhost: Bing virtual hosts search

* virustotal: virustotal.com domain search

* yahoo: Yahoo search engine


Active:
-------
* DNS brute force: dictionary brute force enumeration


Modules that require an API key:
--------------------------------
Add your keys to api-keys.yaml

* bing
* github
* hunter
* intelx
* securityTrails
* shodan
* spyse


Install and dependencies:
-------------------------
* Python 3.7+
* python3 -m pip install pipenv
* https://github.com/laramies/theHarvester/wiki/Installation


Comments, bugs and requests:
----------------------------
* [![Twitter Follow](https://img.shields.io/twitter/follow/laramies.svg?style=social&label=Follow)](https://twitter.com/laramies) Christian Martorella @laramies
cmartorella@edge-security.com
* [![Twitter Follow](https://img.shields.io/twitter/follow/NotoriousRebel1.svg?style=social&label=Follow)](https://twitter.com/NotoriousRebel1) Matthew Brown @NotoriousRebel1
* [![Twitter Follow](https://img.shields.io/twitter/follow/jay_townsend1.svg?style=social&label=Follow)](https://twitter.com/jay_townsend1) Jay "L1ghtn1ng" Townsend @jay_townsend1


Main contributors:
------------------
* [![Twitter Follow](https://img.shields.io/twitter/follow/NotoriousRebel1.svg?style=social&label=Follow)](https://twitter.com/NotoriousRebel1) Matthew Brown @NotoriousRebel1
* [![Twitter Follow](https://img.shields.io/twitter/follow/jay_townsend1.svg?style=social&label=Follow)](https://twitter.com/jay_townsend1) Jay "L1ghtn1ng" Townsend @jay_townsend1
* [![Twitter Follow](https://img.shields.io/twitter/follow/discoverscripts.svg?style=social&label=Follow)](https://twitter.com/discoverscripts) Lee Baird @discoverscripts 
* [![LinkedIn](https://static.licdn.com/scds/common/u/img/webpromo/btn_viewmy_160x25.png)](https://www.linkedin.com/in/janoszold/)  Janos Zold


Thanks:
-------
* John Matherly - Shodan project
* Ahmed Aboul Ela - subdomain names dictionaries (big and small)
New logo :) 2019-09-11 05:08:37 +08:00			`![theHarvester](https://github.com/laramies/theHarvester/blob/master/theHarvester-logo.png)`
Update README.md 2019-09-11 05:18:09 +08:00
PEP8 fixes and LGTM fixes, add flake8 to CI 2019-08-19 06:14:24 +08:00			`[![Build Status](https://travis-ci.com/laramies/theHarvester.svg?branch=master)](https://travis-ci.com/laramies/theHarvester) [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/laramies/theHarvester.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/laramies/theHarvester/context:python)`
add rawsec and other fixes 2019-09-01 09:45:43 +08:00			`[![Rawsec's CyberSecurity Inventory](https://inventory.rawsec.ml/img/badges/Rawsec-inventoried-FF5050_flat_without_logo.svg)](https://inventory.rawsec.ml/)`
Initial commit for version 2.0 2011-05-04 23:07:06 +08:00
			`What is this?`
			`-------------`
Update README.md 2020-03-11 11:42:37 +08:00			`theHarvester is a very simple to use, yet powerful and effective tool designed to be used in the early statges of a<br>`
			`penetration test or red team engagement. Use it for open source intelligence (OSINT) gathering to help determine a<br>`
			`company's external threat landscape on the internet. The tool gathers emails, names, subdomains, IPs and URLs using<br>`
			`multiple public data sources that include:`
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`Passive:`
			`--------`
Updated Twitter accounts. 2019-03-18 12:05:29 +08:00			`* baidu: Baidu search engine - www.baidu.com`
Initial commit for version 2.0 2011-05-04 23:07:06 +08:00
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`* bing: Microsoft search engine - www.bing.com`
Initial commit for version 2.0 2011-05-04 23:07:06 +08:00
Grammar 2019-10-15 06:13:10 +08:00			`* bingapi: Microsoft search engine, through the API (Requires an API key, see below.)`
Local DB, Threatcrowd, output colors. 2018-03-23 06:40:41 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`* Bufferoverun: Uses data from Rapid7's Project Sonar - www.rapid7.com/research/project-sonar/`
Working module 2020-02-12 05:04:29 +08:00
Fix typo in readme 2019-10-20 08:55:38 +08:00			`* CertSpotter: Cert Spotter monitors Certificate Transparency logs - https://sslmate.com/certspotter/`
Add certspotter module 2019-10-14 08:07:16 +08:00
Update readme and deps and fix docker build 2020-03-23 01:36:41 +08:00			`* crtsh: Comodo Certificate search - https://crt.sh/`
Local DB, Threatcrowd, output colors. 2018-03-23 06:40:41 +08:00
Update readme and deps and fix docker build 2020-03-23 01:36:41 +08:00			`* dnsdumpster: DNSdumpster search engine - https://dnsdumpster.com/`
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`* dogpile: Dogpile search engine - www.dogpile.com`
Cleaning Readme 2014-12-17 07:33:13 +08:00
Update readme 2019-09-03 07:46:28 +08:00			`* duckduckgo: DuckDuckGo search engine - www.duckduckgo.com`

Minor changes. 2020-03-11 11:39:21 +08:00			`* Exalead: a Meta search engine - www.exalead.com/search`
Added new module. 2019-01-02 10:43:42 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`* github-code: GitHub code search engine (Requires a GitHub Personal Access Token, see below.) - www.github.com`
WIP: Adding github-code search 2019-06-10 06:01:56 +08:00
Cleaned up README. 2018-12-27 15:19:13 +08:00			`* google: Google search engine (Optional Google dorking.) - www.google.com`
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Grammar 2019-10-15 06:13:10 +08:00			`* hunter: Hunter search engine (Requires an API key, see below.) - www.hunter.io`
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Grammar 2019-10-15 06:13:10 +08:00			`* intelx: Intelx search engine (Requires an API key, see below.) - www.intelx.io`
Updated README.md 2019-02-01 05:13:37 +08:00
Grammar 2019-10-15 06:13:10 +08:00			`* linkedin: Google search engine, specific search for LinkedIn users - www.linkedin.com`
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Grammar 2019-10-15 06:13:10 +08:00			`* netcraft: Internet Security and Data Mining - www.netcraft.com`
Cleaning Readme 2014-12-17 07:33:13 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`* otx: AlienVault Open Threat Exchange - otx.alienvault.com`
Update README.md Add otx to the readme 2019-09-26 06:12:24 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`* securityTrails: Security Trails search engine, the world's largest repository of historical DNS data<br>`
			`(Requires an API key, see below.) - www.securitytrails.com`
Added securitytrailssearch.py to init.py in discovery directory and added more info to README.md 2018-12-23 09:20:55 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`* shodan: Shodan search engine, will search for ports and banners from discovered hosts - www.shodanhq.com`
Cleaning Readme 2014-12-17 07:33:13 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`* Spyse: Web research tools for professionals (Requires an API key.) - spyse.com`
Add WIP Spyse module and fix a lgtm issue in the duckduckgo module Signed-off-by: Jay Townsend <townsend891@hotmail.com> 2019-10-02 07:31:26 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`* Suip: Web research tools that can take over 10 minutes to run, but worth the wait - suip.biz`
Disable my spyse module for now while I wait for the v2 api and put a message to say tyhat suip module can take a while and add it to the readme 2019-10-13 04:13:33 +08:00
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`* threatcrowd: Open source threat intelligence - www.threatcrowd.org`
Cleaning Readme 2014-12-17 07:33:13 +08:00
Added securitytrailssearch.py to init.py in discovery directory and added more info to README.md 2018-12-23 09:20:55 +08:00			`* trello: Search trello boards (Uses Google search.)`
2.6 Changelog and new search engines. 2015-05-11 06:37:00 +08:00
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`* twitter: Twitter accounts related to a specific domain (Uses Google search.)`

			`* vhost: Bing virtual hosts search`
Cleaning Readme 2014-12-17 07:33:13 +08:00
Updated Twitter accounts. 2019-03-18 12:05:29 +08:00			`* virustotal: virustotal.com domain search`
Modified hunter search engine so user just specificies key in huntersearch.py! 2018-11-07 06:12:20 +08:00
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`* yahoo: Yahoo search engine`
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Update README.md 2018-11-23 07:33:04 +08:00
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00			`Active:`
			`-------`
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`* DNS brute force: dictionary brute force enumeration`
Update README.md Removed DNS TLD and reverse lookup, until ported to new library. 2019-09-12 07:10:27 +08:00
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`Modules that require an API key:`
			`--------------------------------`
Done the changes that @leebaird mentioned 2019-01-22 04:46:10 +08:00			`Add your keys to api-keys.yaml`
Cleaned up README. 2018-12-27 15:19:13 +08:00
Shortened bing under Modules that require an API key. 2019-11-26 12:18:50 +08:00			`* bing`
WIP: Adding github-code search 2019-06-10 06:01:56 +08:00			`* github`
Updated README and removed stale code. 2019-02-28 11:31:45 +08:00			`* hunter`
			`* intelx`
			`* securityTrails`
			`* shodan`
Add WIP Spyse module and fix a lgtm issue in the duckduckgo module Signed-off-by: Jay Townsend <townsend891@hotmail.com> 2019-10-02 07:31:26 +08:00			`* spyse`
Initial commit for version 2.0 2011-05-04 23:07:06 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00
			`Install and dependencies:`
			`-------------------------`
Update readme to python3.7+ as a requirement 2019-10-03 07:16:21 +08:00			`* Python 3.7+`
Update readme 2019-10-13 06:48:20 +08:00			`* python3 -m pip install pipenv`
Remove 1 test and tidy up a few things and work on sypse module also add exalead to sources and update README.md about how to install theHarvester 2020-01-15 07:23:20 +08:00			`* https://github.com/laramies/theHarvester/wiki/Installation`

Release 3.0 Added Port scanning, Subdomain takeover, Fixed Shodan, improved Dns bruteforce and added SQLite support to store the results. 2018-08-09 03:34:10 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00			`Comments, bugs and requests:`
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`----------------------------`
Tweaked Main contributors again. 2019-03-19 05:26:54 +08:00			`* [![Twitter Follow](https://img.shields.io/twitter/follow/laramies.svg?style=social&label=Follow)](https://twitter.com/laramies) Christian Martorella @laramies`
Add WIP Spyse module and fix a lgtm issue in the duckduckgo module Signed-off-by: Jay Townsend <townsend891@hotmail.com> 2019-10-02 07:31:26 +08:00			`cmartorella@edge-security.com`
			`* [![Twitter Follow](https://img.shields.io/twitter/follow/NotoriousRebel1.svg?style=social&label=Follow)](https://twitter.com/NotoriousRebel1) Matthew Brown @NotoriousRebel1`
			`* [![Twitter Follow](https://img.shields.io/twitter/follow/jay_townsend1.svg?style=social&label=Follow)](https://twitter.com/jay_townsend1) Jay "L1ghtn1ng" Townsend @jay_townsend1`
2.5 Fixed Google searches, and introduced Requests library 2014-12-17 07:25:12 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00
Update README.md Added collaborators 2019-01-09 07:49:26 +08:00			`Main contributors:`
Removed additional google-profiles code. 2019-01-28 09:33:18 +08:00			`------------------`
Update README.md Updated link. 2019-04-13 02:14:56 +08:00			`* [![Twitter Follow](https://img.shields.io/twitter/follow/NotoriousRebel1.svg?style=social&label=Follow)](https://twitter.com/NotoriousRebel1) Matthew Brown @NotoriousRebel1`
Tweaked Main contributors again. 2019-03-19 05:26:54 +08:00			`* [![Twitter Follow](https://img.shields.io/twitter/follow/jay_townsend1.svg?style=social&label=Follow)](https://twitter.com/jay_townsend1) Jay "L1ghtn1ng" Townsend @jay_townsend1`
Tweaked 'Main contributors. 2019-03-19 05:12:01 +08:00			`* [![Twitter Follow](https://img.shields.io/twitter/follow/discoverscripts.svg?style=social&label=Follow)](https://twitter.com/discoverscripts) Lee Baird @discoverscripts`
Add WIP Spyse module and fix a lgtm issue in the duckduckgo module Signed-off-by: Jay Townsend <townsend891@hotmail.com> 2019-10-02 07:31:26 +08:00			`* [![LinkedIn](https://static.licdn.com/scds/common/u/img/webpromo/btn_viewmy_160x25.png)](https://www.linkedin.com/in/janoszold/) Janos Zold`
Update README.md Added collaborators 2019-01-09 07:49:26 +08:00
Minor changes. 2020-03-11 11:39:21 +08:00
Update README.md Added collaborators 2019-01-09 07:49:26 +08:00			`Thanks:`
			`-------`
Update README.md 2019-01-01 19:58:14 +08:00			`* John Matherly - Shodan project`
Minor Lee fixes 2018-12-23 04:29:11 +08:00			`* Ahmed Aboul Ela - subdomain names dictionaries (big and small)`