gravitl/netmaker
1
1
Fork 0
mirror of https://github.com/gravitl/netmaker.git synced 2025-02-23 23:53:58 +08:00
Code Issues Projects Releases Packages Wiki Activity

fixed startup logs. Removed comms from network list

Browse source
This commit is contained in:
afeiszli 2021-05-30 11:26:10 -04:00
parent 8dfd81794d
commit b3655d2fdf
7 changed files with 121 additions and 212 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

144
README.md
View file

@ -6,139 +6,41 @@
<i>Connect any computers together over a secure, fast, private network, and manage multiple networks from a central server.</i>
</p>
## Documentation
### Netmaker's documentation can be found on ReadTheDocs and on Netmaker.org.
- netmaker.readthedocs.io
- docs.netmaker.org
Below is a quick summary of Netmaker. For information about installing and using Netmaker, please visit the documentation sites.
## What is Netmaker?
Netmaker is a tool for creating and managing virtual networks. If you have servers spread across multiple locations, data centers, or clouds, they all live on separate networks. This can make life very difficult. Netmaker takes all those machines and puts them on a single, flat network so that they can talk to each other easily and securely.
Netmaker is a platform for creating and managing networks that are:
Think of it like Tailscale, ZeroTier, or Nebula, but faster, easier, and more dynamic.
- Fast
- Secure
- Dynamic
- Virtual
- Zero-Trust
- WireGuard-based
You spin up the Netmaker server and UI, and then install the Netclient (agent) on your computers. Netmaker will do the rest. It will tell all of your computers how to reach each other and will keep them informed of any changes to the network.
Netmaker's handy dandy UI can be found [here](https://github.com/gravitl/netmaker-ui).
Under the hood, Netmaker uses WireGuard to create encrypted tunnels between every node in your virtual network, creating a full mesh overlay. Netmaker takes the work out of manually configuring machines with WireGuard and updating them every time you have a change in your network. The netclient agent is self-updating and pulls any necessary changes (such as new peers) from the server.
Netmaker manages networks between computers that span data centers, clouds, and edge devices. Networking can be a challenge, so Netmaker is designed to allow developers, devops engineers, SRE's, and more to manage virtual networks with as little pain as possible. You create a network with a few clicks, and add machines to that network with a single command.
## Why Netmaker?
1. Create a flat, secure network between multiple/hybrid cloud environments
2. Integrate central and edge services
3. Secure a home or office network while providing remote connectivity
4. Manage cryptocurrency proof-of-stake machines
6. Provide an additional layer of security on an existing network
7. Encrypt Kubernetes inter-node communications
8. Secure site-to-site connections
4. Provide an additional layer of security on an existing network
5. Encrypt Kubernetes inter-node communications
6. Secure site-to-site connections
7. Connect to IoT devices
8. Manage an existing WireGuard setup with less manual configuration
<p align="center">
<img src="mesh-diagram.png">
</p>
## Compatible Systems
Netmaker is primarily designed for **linux**, specifically **systemd-based linux.** This includes Fedora, Ubuntu, and Raspian. Just make sure you have WireGuard installed. Having a problem? Open an issue or Contact us.
In version 0.3 we have released Private DNS. Nameservers can be configured manually on any system, but to have the Netclient add dns automatically, it requires **resolvectl.**
In future releases, we have plans to support other platforms such as Windows and MacOS.
## Docs
**For more information, please read the docs, or check out the Quick Start below:**
- [General Usage](docs/USAGE.md)
- [Troubleshooting](docs/TROUBLESHOOTING.md)
- [API Documentation](docs/API.md)
- [Product Roadmap](docs/ROADMAP.md)
- [Contributing](docs/CONTRIBUTING.md)
## Quick Start
[Intro/Overview Video Tutorial](https://youtu.be/PWLPT320Ybo)
[Site-to-Site Video Tutorial](https://youtu.be/krCKBJhwwDk)
### Note about permissions
The default installation requires special privileges on the server side, because Netmaker will control the local kernel Wireguard. This can be turned off and run in non-privileged mode if necessary (but disables some features). For more details, see the **Usage** docs.
### Prereqs
1. A running linux server to host Netmaker, with an IP reachable by your computers (Debian-based preferred but not required).
2. Linux installed on the above server (Debian-based preferred but not required).
3. Install Docker and Docker Compose if running in Docker Mode (see below).
4. System dependencies installed:
- Docker (if running in default Docker mode. DO NOT use snap install for docker.)
- Docker Compose
- Wireguard + Resolvectl (if running in default Client mode)
#### CoreDNS Preparation
v0.3 introduces CoreDNS as a private nameserver. To run CoreDNS on your server host, you must disable systemd-resolved to open port 53:
1. systemctl stop systemd-resolved
2. systemctl disable systemd-resolved
3. vim /etc/systemd/resolved.conf
- uncomment **DNS=** and add 8.8.8.8 or whatever is your preference
- uncomment **DNSStubListener=** and set to **"no"**
4. sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
### Launch Netmaker
Note, this installs Netmaker with CoreDNS and a Netclient (privileged). If you want to run the server non-privileged or without CoreDNS, see the advanced usage docs.
1. Clone this repo or just copy contents of "docker-compose.yml" to your Netmaker server (from prereqs).
2. In docker-compose.yml, change BACKEND_URL to the public IP of your server.
3. Run `sudo docker-compose up -d`
4. Navigate to your server's IP in the browser and you should see the Netmaker UI asking to create a new admin user.
5. Create a new admin user
6. You are now ready to begin using Netmaker.
### Create a Network
You can also just use the "default" network.
1. Click "CREATE NETWORK" in the upper left of your console
2. Enter a valid address range, e.g. 10.11.12.0/24
3. Enter a name such as "homenet"
4. Additional options:
- **Dual Stack**: Machines will recieve a private IPv6 address in addition to their IPv4 address.
- **Local:** Will use local address range for endpoints instead of public. Use Case: Home or Office network where most devices do not have public IP's. In this case you can create a gateway into the network after creating the Local Network.
After Network creation, you can edit the network in the NETWORK DETAILS pane, modifying the address range and default options. You can also toggle on **Allow Node Signup Without Keys**, which makes the next step unnecessary, but allows anyone to create a node in your network, which will be cordoned in pending state.
### Create Keys
1. Click the "ACCESS KEYS" tab
2. Click "ADD NEW ACCESSS KEY"
3. Give your key a name and number of uses
4. Several values will be displayed. Save these somewhere, as they will only be displayed once:
- **Access Key:** Use only in special edge cases where server connection string must be modified
- **Access Token:** Use on machines that already have the netclient utility
- **Install Command:** Use on machines that do not have the netclient utility
### Install Agent:
For machines **without** netclient, run the install command (from above): `curl -sfL https://raw.githubusercontent.com/gravitl/netmaker/v0.3/netclient-install.sh | KEY=<your access key> sh -`
For machines **with** netclient run the following (with access token from above): `sudo netclient -c install -t <access token>`
For networks with **manual signup** enabled (see above), install using the network name: `sudo netclient -c install -n <network name>`
### Manage Nodes
Your machines should now be visible in the control pane.
**Modify nodes:** Click the pencil icon in the NODES pane to modify details like WireGuard port, address, and node name. You can also **DELETE** nodes here and they will lose network access.
**Approve nodes:** If a node is in pending state (signed up without key), you can approve it. An icon will appear for pending nodes that need approval.
**Gateway Mode:** Click the Gateway icon to enable gateway mode on a given node. A popup will allow you to choose an existing network, or enter a custom address range.
*Example: You create a network in netmaker called Homenet. It has several machines on your home server. You create another network called Cloudnet. It has several machines in AWS. You have one server (server X) which is added to both networks. On Cloudnet, you make Server X a gateway to Homenet. Now, the cloudnet machines have access to your homenet machines. via Server X.*
*On Homenet, you add Server Y, a machine in AWS, and make it a gateway to a custom address range 172.16.0.0/16. The machines on your home network now have access to any AWS machines in that address range via Server Y*
### Manage DNS
On the DNS tab you can create custom DNS entries for a given network.
1. All dns entries will be *postfixed* with a private TLD of the network name, for example, ".mynet"
2. Default DNS is created for node name + TLD, for instance, node-c42wt.mynet. This is not editable.
3. Click ADD ENTRY to add custom DNS
- You can click CHOOSE NODE to direct DNS to a specific node in the network
- You can also specify any custom address you would like, which can be outside the network (for instance, the IP for google.com)
- Add a dns entry name, which will be postfixed with the network TLD. E.g. if you enter "privateapi.com", it will become "privateapi.com.networkname"
### Uninstalling Client
To uninstall the client from a network: `sudo netclient -c remove -n < networkname >`
To uninstall entirely, run the above for each network, and then run `sudo rm -rf /etc/netclient`
### Uninstralling Netmaker
To uninstall the netmaker server, simply run `docker-compose down`
#### LICENSE
@ -146,5 +48,5 @@ Netmaker's source code and all artifacts in this repository are freely available
#### CONTACT
Email: alex@gravitl.com
Email: info@gravitl.com
Discord: https://discord.gg/zRb9Vfhk8A

13
controllers/controller.go
View file

@ -5,7 +5,7 @@ import (
"github.com/gravitl/netmaker/servercfg"
"os/signal"
"os"
"fmt"
"log"
"context"
"net/http"
"github.com/gorilla/mux"
@ -39,13 +39,12 @@ func HandleRESTRequests(wg *sync.WaitGroup) {
srv := &http.Server{Addr: ":" + port, Handler: handlers.CORS(originsOk, headersOk, methodsOk)(r)}
go func(){
err := srv.ListenAndServe()
//err := http.ListenAndServe(":" + port,
//handlers.CORS(originsOk, headersOk, methodsOk)(r))
if err != nil {
fmt.Println(err)
log.Println(err)
}
}()
fmt.Println("REST Server succesfully started on port " + port + " (REST)")
log.Println("REST Server succesfully started on port " + port + " (REST)")
c := make(chan os.Signal)
// Relay os.Interrupt to our channel (os.Interrupt = CTRL+C)
@ -57,8 +56,8 @@ func HandleRESTRequests(wg *sync.WaitGroup) {
<-c
// After receiving CTRL+C Properly stop the server
fmt.Println("Stopping the REST server...")
log.Println("Stopping the REST server...")
srv.Shutdown(context.TODO())
fmt.Println("REST Server closed.")
log.Println("REST Server closed.")
mongoconn.Client.Disconnect(context.TODO())
}

27
controllers/networkHttpController.go
View file

@ -97,16 +97,31 @@ func authenticateMaster(tokenString string) bool {
//simple get all networks function
func getNetworks(w http.ResponseWriter, r *http.Request) {
networks, err := functions.ListNetworks()
allnetworks, err := functions.ListNetworks()
if err != nil {
returnErrorResponse(w, r, formatError(err, "internal"))
return
} else {
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(networks)
return
}
networks := RemoveComms(allnetworks)
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(networks)
return
}
func RemoveComms(networks []models.Network) []models.Network {
var index int = 100000001
for ind, net := range networks {
if net.NetID == "comms" {
index = ind
}
}
if index == 100000001 {
return networks
}
returnable := make([]models.Network, 0)
returnable = append(returnable, networks[:index]...)
return append(returnable, networks[index+1:]...)
}
func ValidateNetworkUpdate(network models.NetworkUpdate) error {

23
functions/local.go
View file

@ -1,7 +1,6 @@
package functions
import (
"fmt"
"io/ioutil"
"log"
"os"
@ -15,12 +14,22 @@ func FileExists(f string) bool {
return !info.IsDir()
}
func SetDNSDir() error {
dir, err := os.Getwd()
if err != nil {
return err
}
_, err = os.Stat(dir + "/config/dnsconfig")
if os.IsNotExist(err) {
os.Mkdir(dir+"/config/dnsconfig", 744)
} else if err != nil {
log.Println("couldnt find or create /config/dnsconfig")
return err
}
return nil
}
func SetCorefile(domains string) error {
//does not work when executing tests
//dir, err := filepath.Abs(filepath.Dir(os.Args[0]))
//if err != nil {
// return err
//}
dir, err := os.Getwd()
if err != nil {
return err
@ -29,7 +38,7 @@ func SetCorefile(domains string) error {
if os.IsNotExist(err) {
os.Mkdir(dir+"/config/dnsconfig", 744)
} else if err != nil {
fmt.Println("couldnt find or create /config/dnsconfig")
log.Println("couldnt find or create /config/dnsconfig")
return err
}

59
main.go
View file

@ -26,6 +26,7 @@ import (
//Start MongoDB Connection and start API Request Handler
func main() {
//Client Mode Prereq Check
if servercfg.IsClientMode() {
cmd := exec.Command("id", "-u")
@ -44,6 +45,12 @@ func main() {
log.Fatal("To run in client mode requires root privileges. Either disable client mode or run with sudo.")
}
}
if servercfg.IsDNSMode() {
err := functions.SetDNSDir()
if err != nil {
log.Fatal(err)
}
}
//Start Mongodb
mongoconn.ConnectDatabase()
@ -53,32 +60,19 @@ func main() {
//Create the default network (default: 10.10.10.0/24)
created, err := serverctl.CreateDefaultNetwork()
if err != nil {
fmt.Printf("Error creating default network: %v", err)
log.Printf("Error creating default network: %v", err)
}
if created && servercfg.IsClientMode() {
installserver = true
}
if servercfg.IsGRPCWireGuard() {
exists, err := functions.ServerIntClientExists()
if err == nil {
if !exists {
err = serverctl.InitServerWireGuard()
if err != nil {
log.Fatal(err)
}
}
}
err = serverctl.ReconfigureServerWireGuard()
err = serverctl.InitServerWireGuard()
//err = serverctl.ReconfigureServerWireGuard()
if err != nil {
log.Fatal(err)
}
}
//NOTE: Removed Check and Logic for DNS Mode
//Reasoning. DNS Logic is very small on server. Can run with little/no impact. Just sets a tiny config file.
//Real work is done by CoreDNS
//We can just not run CoreDNS. On Agent side is only necessary check for IsDNSMode, which we will pass.
var waitnetwork sync.WaitGroup
@ -87,7 +81,7 @@ func main() {
if !(servercfg.DisableRemoteIPCheck()) && servercfg.GetGRPCHost() == "127.0.0.1" {
err := servercfg.SetHost()
if err != nil {
fmt.Println("Unable to Set host. Exiting.")
log.Println("Unable to Set host. Exiting...")
log.Fatal(err)
}
}
@ -100,7 +94,7 @@ func main() {
if !servercfg.DisableRemoteIPCheck() && servercfg.GetAPIHost() == "127.0.0.1" {
err := servercfg.SetHost()
if err != nil {
fmt.Println("Unable to Set host. Exiting.")
log.Println("Unable to Set host. Exiting...")
log.Fatal(err)
}
}
@ -108,10 +102,10 @@ func main() {
controller.HandleRESTRequests(&waitnetwork)
}
if !servercfg.IsAgentBackend() && !servercfg.IsRestBackend() {
fmt.Println("Oops! No Server Mode selected. Nothing is being served! Set either Agent mode (AGENT_BACKEND) or Rest mode (REST_BACKEND) to 'true'.")
log.Println("No Server Mode selected, so nothing is being served! Set either Agent mode (AGENT_BACKEND) or Rest mode (REST_BACKEND) to 'true'.")
}
waitnetwork.Wait()
fmt.Println("Exiting now.")
log.Println("exiting")
}
@ -150,24 +144,23 @@ func runGRPC(wg *sync.WaitGroup, installserver bool) {
log.Fatalf("Failed to serve: %v", err)
}
}()
fmt.Println("Agent Server succesfully started on port " + grpcport + " (gRPC)")
log.Println("Agent Server succesfully started on port " + grpcport + " (gRPC)")
if installserver {
fmt.Println("Adding server to default network")
success, err := serverctl.AddNetwork("default")
if err != nil {
fmt.Printf("Error adding to default network: %v", err)
fmt.Println("")
fmt.Println("Unable to add server to network. Continuing.")
fmt.Println("Please investigate client installation on server.")
log.Printf("Error adding to default network: %v", err)
log.Println("Unable to add server to network. Continuing.")
log.Println("Please investigate client installation on server.")
} else if !success {
fmt.Println("Unable to add server to network. Continuing.")
fmt.Println("Please investigate client installation on server.")
log.Println("Unable to add server to network. Continuing.")
log.Println("Please investigate client installation on server.")
} else{
fmt.Println("Server successfully added to default network.")
log.Println("Server successfully added to default network.")
}
}
fmt.Println("Setup complete. You are ready to begin using netmaker.")
log.Println("Setup complete. You are ready to begin using netmaker.")
// Right way to stop the server using a SHUTDOWN HOOK
// Create a channel to receive OS signals
@ -182,13 +175,13 @@ func runGRPC(wg *sync.WaitGroup, installserver bool) {
<-c
// After receiving CTRL+C Properly stop the server
fmt.Println("Stopping the Agent server...")
log.Println("Stopping the Agent server...")
s.Stop()
listener.Close()
fmt.Println("Agent server closed..")
fmt.Println("Closing MongoDB connection")
log.Println("Agent server closed..")
log.Println("Closing MongoDB connection")
mongoconn.Client.Disconnect(context.TODO())
fmt.Println("MongoDB connection closed.")
log.Println("MongoDB connection closed.")
}
func authServerUnaryInterceptor() grpc.ServerOption {

51
serverctl/serverctl.go
View file

@ -1,7 +1,7 @@
package serverctl
import (
"fmt"
"log"
"github.com/gravitl/netmaker/functions"
"github.com/gravitl/netmaker/models"
"github.com/gravitl/netmaker/mongoconn"
@ -18,13 +18,13 @@ import (
func CreateDefaultNetwork() (bool, error) {
fmt.Println("Creating default network...")
log.Println("Creating default network...")
iscreated := false
exists, err := functions.NetworkExists("default")
if exists || err != nil {
fmt.Println("Default network already exists. Skipping...")
log.Println("Default network already exists. Skipping...")
return iscreated, err
} else {
@ -43,8 +43,6 @@ func CreateDefaultNetwork() (bool, error) {
allow := true
network.AllowManualSignUp = &allow
fmt.Println("Creating default network.")
collection := mongoconn.Client.Database("netmaker").Collection("networks")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
@ -56,6 +54,7 @@ func CreateDefaultNetwork() (bool, error) {
if err == nil {
iscreated = true
}
log.Println("1")
return iscreated, err
@ -75,13 +74,11 @@ func GetServerWGConf() (models.IntClient, error) {
func CreateCommsNetwork() (bool, error) {
fmt.Println("Creating GRPC network...")
iscreated := false
exists, err := functions.NetworkExists("comms")
if exists || err != nil {
fmt.Println("GRPC network already exists. Skipping...")
log.Println("comms network already exists. Skipping...")
return true, nil
} else {
@ -101,7 +98,7 @@ func CreateCommsNetwork() (bool, error) {
network.IsLocal = &priv
network.KeyUpdateTimeStamp = time.Now().Unix()
fmt.Println("Creating comms network.")
log.Println("Creating comms network...")
collection := mongoconn.Client.Database("netmaker").Collection("networks")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
@ -122,7 +119,7 @@ func DownloadNetclient() error {
// Get the data
resp, err := http.Get("https://github.com/gravitl/netmaker/releases/download/latest/netclient")
if err != nil {
fmt.Println("could not download netclient")
log.Println("could not download netclient")
return err
}
defer resp.Body.Close()
@ -133,7 +130,7 @@ func DownloadNetclient() error {
if !FileExists("/etc/netclient/netclient") {
_, err := copy("./netclient/netclient", "/etc/netclient/netclient")
if err != nil {
fmt.Println("could not create /etc/netclient")
log.Println("could not create /etc/netclient")
return err
}
}
@ -159,7 +156,7 @@ func copy(src, dst string) (int64, error) {
}
if !sourceFileStat.Mode().IsRegular() {
return 0, fmt.Errorf("%s is not a regular file", src)
return 0, errors.New(src + " is not a regular file")
}
source, err := os.Open(src)
@ -176,7 +173,7 @@ func copy(src, dst string) (int64, error) {
nBytes, err := io.Copy(destination, source)
err = os.Chmod(dst, 0755)
if err != nil {
fmt.Println(err)
log.Println(err)
}
return nBytes, err
}
@ -184,15 +181,15 @@ func copy(src, dst string) (int64, error) {
func RemoveNetwork(network string) (bool, error) {
_, err := os.Stat("/etc/netclient/netclient")
if err != nil {
fmt.Println("could not find /etc/netclient")
log.Println("could not find /etc/netclient")
return false, err
}
cmdoutput, err := exec.Command("/etc/netclient/netclient","-c","remove","-n",network).Output()
if err != nil {
fmt.Println(string(cmdoutput))
log.Println(string(cmdoutput))
return false, err
}
fmt.Println("Server removed from network " + network)
log.Println("Server removed from network " + network)
return true, err
}
@ -200,7 +197,7 @@ func RemoveNetwork(network string) (bool, error) {
func AddNetwork(network string) (bool, error) {
pubip, err := servercfg.GetPublicIP()
if err != nil {
fmt.Println("could not get public IP.")
log.Println("could not get public IP.")
return false, err
}
@ -208,36 +205,36 @@ func AddNetwork(network string) (bool, error) {
if os.IsNotExist(err) {
os.Mkdir("/etc/netclient", 744)
} else if err != nil {
fmt.Println("could not find or create /etc/netclient")
log.Println("could not find or create /etc/netclient")
return false, err
}
fmt.Println("Directory is ready.")
}
log.Println("Directory is ready.")
token, err := functions.CreateServerToken(network)
if err != nil {
fmt.Println("could not create server token for " + network)
log.Println("could not create server token for " + network)
return false, err
}
fmt.Println("Token is ready.")
log.Println("Token is ready.")
_, err = os.Stat("/etc/netclient/netclient")
if os.IsNotExist(err) {
err = DownloadNetclient()
fmt.Println("could not download netclient")
log.Println("could not download netclient")
if err != nil {
return false, err
}
}
err = os.Chmod("/etc/netclient/netclient", 0755)
if err != nil {
fmt.Println("could not change netclient directory permissions")
log.Println("could not change netclient directory permissions")
return false, err
}
fmt.Println("Client is ready. Running install.")
log.Println("Client is ready. Running install.")
out, err := exec.Command("/etc/netclient/netclient","-c","install","-t",token,"-name","netmaker","-ip4",pubip).Output()
fmt.Println(string(out))
log.Println(string(out))
if err != nil {
return false, errors.New(string(out) + err.Error())
}
fmt.Println("Server added to network " + network)
log.Println("Server added to network " + network)
return true, err
}

16
serverctl/wireguard.go
View file

@ -1,6 +1,7 @@
package serverctl
import (
//"github.com/davecgh/go-spew/spew"
"os"
"log"
"context"
@ -38,8 +39,7 @@ func InitServerWireGuard() error {
err = netlink.LinkAdd(wglink)
if err != nil {
if os.IsExist(err) {
log.Println("interface " + ifaceSettings.Name + " already exists")
log.Println("continuing setup using existing interface")
log.Println("WireGuard interface " + ifaceSettings.Name + " already exists. Skipping...")
} else {
return err
}
@ -50,13 +50,8 @@ func InitServerWireGuard() error {
}
err = netlink.AddrAdd(wglink, address)
if err != nil {
if os.IsExist(err) {
log.Println("address " + wgconfig.GRPCWGAddress + " already exists")
log.Println("continuing with existing setup")
} else {
if err != nil && !os.IsExist(err){
return err
}
}
err = netlink.LinkSetUp(wglink)
if err != nil {
@ -138,7 +133,6 @@ func ReconfigureServerWireGuard() error {
if err != nil {
return err
}
wgserver, err := wgctrl.New()
if err != nil {
return err
@ -180,10 +174,10 @@ func ReconfigureServerWireGuard() error {
ReplacePeers: true,
Peers: serverpeers,
}
err = wgserver.ConfigureDevice(servercfg.GetGRPCWGInterface(), wgconf)
wgiface := servercfg.GetGRPCWGInterface()
err = wgserver.ConfigureDevice(wgiface, wgconf)
if err != nil {
return err
}
return nil
}