Commit graph

337 commits

Author SHA1 Message Date
Abhishek K
b972e7a969
add mutex on failover ops (#3590) 2025-08-11 12:17:11 +05:30
Abhishek K
aca911712b
avoid setting nil endpoint if peer using internet gw (#3529) 2025-06-25 19:17:57 +05:30
abhishek9686
b423b183e4 fix merge conflicts 2025-06-24 16:56:48 +05:30
Abhishek K
0a47cc5461
Merge branch 'develop' into NET-1994 2025-06-23 17:50:21 +05:30
abhishek9686
8ee37f5fab inet gws into gateways 2025-06-17 16:55:44 +05:30
abhishek9686
9760c0945f ignore last checkin if time is zero 2025-06-16 13:46:44 +05:30
abhishek9686
0761e73e43 collection loc info for desktop config if unset 2025-06-13 23:29:36 +05:30
abhishek9686
fcbb524b82 check if node is egress gw 2025-06-09 18:00:38 +05:30
abhishek9686
6be6304a64 fix host dns setting 2025-06-09 17:17:26 +05:30
Abhishek K
adc4d7f3dd
NET-2061: fix egress policies (#3485)
* revert inet gws from acl policies

* add egress range with metric for inet gw

* link pro inet funcs

* fix extclient comms with users

* remove TODO comments

* add backwards compatibility to egress ranges

* remove all resources check

* remove device policy check on pro

* fix egress policies for users groups

* add default forwarding rule for inet gw
2025-06-06 13:10:36 +05:30
Abhishek K
fdc8ea4320
NET-2061: Fix egress user policies (#3484)
* revert inet gws from acl policies

* add egress range with metric for inet gw

* link pro inet funcs

* fix extclient comms with users

* remove TODO comments

* add backwards compatibility to egress ranges

* remove all resources check

* remove device policy check on pro
2025-06-06 07:36:49 +05:30
Abhishek K
0f884d4f36
NET-2061: revert Inet gws, fix extclient comms with user policies (#3482)
* revert inet gws from acl policies

* add egress range with metric for inet gw

* link pro inet funcs

* fix extclient comms with users

* remove TODO comments
2025-06-05 22:42:16 +05:30
Abhishek K
0ff216dabd
NET-1932: Fix static node comms on CE (#3481)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

* fix default network user policy

* fix egress migration

* fix egress migration

* add failover inet gw check

* optiomise egress calls

* auto create gw on inet egress node

* optimise egress calls

* add global user role check

* fix egress on inet gw

* remove addtional checks on inet policy

* add acl policy for static nodes on CE

* remove chained inet gws

* fix multi-inet issue

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-05 13:28:23 +05:30
Abhishek K
44300590f8
NET-1932: remove addtional checks on Inet policy, optimise acl calls (#3480)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

* fix default network user policy

* fix egress migration

* fix egress migration

* add failover inet gw check

* optiomise egress calls

* auto create gw on inet egress node

* optimise egress calls

* add global user role check

* fix egress on inet gw

* remove addtional checks on inet policy

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-04 12:45:46 +05:30
Abhishek K
599a9c6f4a
NET-1932: handle non-inet egress ranges on inet gw (#3479)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

* fix default network user policy

* fix egress migration

* fix egress migration

* add failover inet gw check

* optiomise egress calls

* auto create gw on inet egress node

* optimise egress calls

* add global user role check

* fix egress on inet gw

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-04 08:04:40 +05:30
Abhishek K
3bae08797f
NET-1932: add failover inet gw check (#3478)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

* fix default network user policy

* fix egress migration

* fix egress migration

* add failover inet gw check

* optiomise egress calls

* auto create gw on inet egress node

* optimise egress calls

* add global user role check

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-03 22:54:55 +05:30
Abhishek K
5849e0e2e1
NET-1932: sort out acls into CE and PRO (#3460)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-02 13:05:10 +05:30
Abhishek K
a1304b43d8
NET-2054: Auto Removal of Offline Nodes, fix enrollment key relay function (#3458)
* check host ports on join

* if 443 not available fallback to 51821

* if 443 not available fallback to 51821

* add config for auto delete of offline nodes

* autocleanup offline nodes

* delete offline nodes on startup

* fix relay via join token
2025-05-24 08:21:47 +05:30
Abhishek K
307a3d1e4b
NET-1932: Merge egress and internet gateways (#3436)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* egress model

* fix revoked tokens to be unauthorized

* update egress model

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* setup api handlers for egress revamp

* use single DB, fix update nat boolean field

* extend validaiton checks for egress ranges

* add migration to convert to new egress model

* fix panic interface conversion

* publish peer update on settings update

* revoke token generated by an user

* add user token creation restriction by user role

* add forbidden check for access token creation

* revoke user token when group or role is changed

* add default group to admin users on update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* remove nat check on egress gateway request

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* add inet gw status to egress model

* fetch node ids in the tag, add inet gw info clients

* add inet gw info to node from egress list

* add migration logic internet gws

* create default acl policies

* add egress info

* add egress TODO

* add egress TODO

* fix user auth api:

* add reference id to acl policy

* add egress response from DB

* publish peer update on egress changes

* re initalise oauth and email config

* set verbosity

* normalise cidr on egress req

* add egress id to acl group

* change acls to use egress id

* resolve merge conflicts

* fix egress reference errors

* move egress model to schema

* add api context to DB

* sync auto update settings with hosts

* sync auto update settings with hosts

* check acl for egress node

* check for egress policy in the acl dst groups

* fix acl rules for egress policies with new models

* add status to egress model

* fix inet node func

* mask secret and convert jwt duration to minutes

* enable egress policies on creation

* convert jwt duration to minutes

* add relevant ranges to inet egress

* skip non active egress routes

* resolve merge conflicts

* fix static check

* update gorm tag for primary key on egress model

* create user policies for egress resources

* resolve merge conflicts

* get egress info on failover apis, add egress src validation for inet gws

* add additional validation checks on egress req

* add additional validation checks on egress req

* skip all resources for inet policy

* delete associated egress acl policies

* fix failover of inetclient

* avoid setting inet client asd inet gw

* fix all resource egress policy

* fix inet gw egress rule

* check for node egress on relay req

* fix egress acl rules comms

* add new field for egress info on node

* check acl policy in failover ctx

* avoid default host to be set as inet client

* fix relayed egress node

* add valid error messaging for egress validate func

* return if inet default host

* jump port detection to 51821

* check host ports on pull

* check user access gws via acls

* add validation check for default host and failover for inet clients

* add error messaging for acl policy check

* fix inet gw status

* ignore failover req for peer using inet gw

* check for allowed egress ranges for a peer

* add egress routes to static nodes by access

* avoid setting failvoer as inet client

* fix egress error messaging

* fix extclients egress comms

* fix inet gw acting as inet client

* return formatted error on update acl validation

* add default route for static nodes on inetclient

* check relay node acting as inetclient

* move inet node info to separate field, fix all resouces policy

* remove debug logs

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-05-21 12:50:21 +05:30
Abhishek K
309e4795a1
NET-1950: Persist Server Settings in the DB (#3419)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* publish peer update on settings update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* set verbosity

* sync auto update settings with hosts

* sync auto update settings with hosts

* mask secret and convert jwt duration to minutes

* convert jwt duration to minutes

* notify peers after settings update

* compare with curr settings before updating

* send host update to devices on auto update

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-04-30 02:34:10 +04:00
Abhishek K
e2a0ceccf6
fix egress policies acls comms (#3420) 2025-04-14 15:21:06 +04:00
Abhishek K
5e1ef5e366
Net 1990 (#3385)
* add peerkey to network egress routes model

* add peerkey to network egress routes model

* filter out conflicting routes from node

* add support for egress HA on relay

* add support for egress HA on relay

* add support for egress HA on relay

* skip if curr node is relay node of the peer

* skip if curr node is relay node of the peer

* fix failover egress HA

* add network to egress route model

* clone before modifying

* check for user policies on uni direction check

* add all network fwd rule
2025-03-25 00:29:35 +04:00
Abhishek K
4cc56fd3be
NET-1990: add peerkey to network egress routes model (#3379)
* add peerkey to network egress routes model

* add peerkey to network egress routes model

* filter out conflicting routes from node

* add support for egress HA on relay

* add support for egress HA on relay

* add support for egress HA on relay

* skip if curr node is relay node of the peer

* skip if curr node is relay node of the peer

* fix failover egress HA

* add network to egress route model

* clone before modifying
2025-03-24 15:33:39 +04:00
Abhishek K
3d765f9cf1
NET-1910: Acl controls for Egress Traffic (#3377)
* add support for egress ranges on acl policy

* add egress ranges to acl rules

* add egress ranges to acl policies

* Add egress ranges to acl rules

* add egress ranges to fw update

* fetch acl rules for egress networks

* apply egress policies for devices

* configure user policies for egresss routes

* fix gw tag name migration

* fix egress acl rules for static nodes

* add egress ranges for static nodes on ingress gw

* fileter acl IPs to be unique

* cleanup IOT logic from peer update

* make acl Rule Dst List

* cleanup egress ranges from acl policies

* create user group default acl policy for gateways

* remove remote access name ids

* rm egress ranges removal from acl policies

* simplify user permissions on nodes

* add additional nameservers to extclient dns

* remove debug logs

* fix static checks
2025-03-18 13:25:55 +04:00
abhishek9686
bb25739bd9 resolve merge conflicts 2025-03-06 22:41:56 +04:00
abhishek9686
f5d6717186 add egress routes with metric to peer update 2025-03-03 00:34:45 +04:00
abhishek9686
33fafa83e6 add nameservers to dns settins 2025-02-28 21:49:53 +04:00
abhishek9686
41fa0b1bce resolve merge conflicts 2025-02-14 18:24:56 +04:00
Abhishek K
689b9b6b52
NET-1941: failover signaling optimized, new endpoint for metrics peer info (#3322)
* optimise acl cache with network map

* fix mutex clash

* fix acl mutex

* add new endpoint for metrics peer info

* fix static check

* add new endpoint for metrics peer info

* fix host peer info api params

* cache failover data

* add json tag

* avoid duplicate calls to failover

* add failover check ctx

* add failover check ctx

* fix failover debug log

* optimise failover operations

* remove debug logs

* rm unused model

* rm unused model
2025-02-14 16:06:26 +04:00
abhishek9686
fabc9f2920 optimise firewall rules 2025-02-08 20:57:09 +04:00
Abhishek K
64bc351564
Merge pull request #3315 from gravitl/master
Master
2025-02-04 19:32:14 +04:00
abhishek9686
01e7ccf731 don't override acl rules for multi-net host 2025-02-04 15:34:03 +04:00
Abhishek K
e13bf2c0eb
NET-1923: Add Metric Port to server config (#3306)
* set default metrics port 8889

* set default metrics port 51821

* add metrics port to server config

* bind caddy only on tcp

* add var for pulling files

* add new line

* update peer update model

* check if port is not zero

* set replace peer to false on pull

* do not replace peers on failover sync

* remove debug log

* add old peer update fields for backwards compatibility

* add old json tag

* add debug log in caller trace func
2025-02-04 08:44:24 +04:00
Abhishek K
8297642b90
NET-1914: add gw apis, move relays into CE (#3309)
* add gw apis, move relays into CE

* set gw field on relay and ingress creation

* add gw handlers to relay and ingress apis

* if node is inetgw and gw add dns

* remove pro check on relays

* fetch node before updating
2025-01-28 11:28:31 +05:30
abhishek9686
56f979bba9 fix peer list on interface 2024-12-16 22:26:09 +04:00
abhishek9686
0216c596cb add all networks rules 2024-12-15 10:41:59 +04:00
abhishek9686
98e313242b add allowed networks 2024-12-13 14:08:34 +04:00
Abhishek K
f124b10c35
NET-1784: Adv Acl Rules (#3239)
* define direction on acl req

* define protocol types and rule model

* get rules for node

* fetch acl rule for a node

* redine acl firewall model

* add json tags

* update port,protocol, and direction

* add json tags to acl options

* convert protocol to string

* simplify acl map

* add json tags to acl rules

* add networks to fw update

* add acls rules

* NET-1784: add allow all field

* add allow all field on fw udpate

* remove debug logs

* fix port and protocol types

* migrate default acl policies

* define constants for service types

* add adv options for user rules on ingress gw

* debug log

* allow whole network

* add static nodes to acl rules

* replace peers on acl updates

* initiliase rule map

* add user acl rules on target node

* revert acl check on extclient

* handle static node rules on ingress gw

* update multiple policies for users

* check allowed direction

* remove debug logs
2024-12-10 11:21:14 +04:00
Yabin Ma
5f21c8bb1d
NET-1778: scale test code changes (#3203)
* comment ACL call and add debug message

* add cache for network nodes

* fix load node to network cache issue

* add peerUpdate call 1 min limit

* add debug log for scale test

* release maps

* avoid default policy for node

* 1 min limit for peerUpdate trigger

* mq options

* Revert "mq options"

This reverts commit 10b93d0118.

* set peerUpdate run in sequence

* update for emqx 5.8.2

* remove batch peer update

* change the sleep to 10 millisec to avoid timeout

* add compress and change encrypt for peerUpdate message

* add mem profiling and automaxprocs

* add failover ctx mutex

* ignore request to failover peer

* remove code without called

* remove debug logs

* update emqx to v5.8.2

* change broker keepalive

* add OLD_ACL_SUPPORT setting

* add host version check for message encrypt

* remove debug message

* remove peerUpdate call control

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
2024-12-10 10:15:31 +04:00
Yabin Ma
87ef555542
NET1847:Add STUN settings (#3235)
* add setting to turn on/off STUN

* sync stun setting in peerUpdate

* sync stun servers setting in peerUpdate
2024-12-06 09:38:32 +04:00
abhishek9686
edcb19f7eb set relayed peer endpoint to nil on relay 2024-11-10 20:28:33 +04:00
abhishek9686
dc03269771 set peer endpoint if host endpoint is nil 2024-11-10 19:14:45 +04:00
abhishek9686
87b790aaa6 set peer endpoint if host endpoint is nil 2024-11-10 18:59:52 +04:00
abhishek9686
9331431a4e Merge branch 'develop' of https://github.com/gravitl/netmaker into NET-1615 2024-10-30 11:40:22 +04:00
Max Ma
5c15f3d9eb
NET-1603: Manage DNS NM changes (#3124)
* add switch for manage dns

* manage DNS sync publish

* add dns sync api

* add manageDNS field in peerUpdate

* add default dns for extClent if manage dns enabled

* add DEFAULT_DOMAIN for internal DNS lookup

* move DNSSync to peerUpdate

* fix empty host in network issue

* sync up dns when custom dns add/delete

* fix custom DNS ip4/ipv6 validator issue
2024-10-29 13:53:45 +04:00
abhishek9686
8e5a3d02c3 set egress6 ranges 2024-10-29 12:56:11 +04:00
abhishek9686
ebc3e90301 add extclient egress ranges 2024-10-29 12:41:40 +04:00
abhishek9686
541bec71de fix default policy check 2024-10-23 15:55:04 +04:00
abhishek9686
d4da1774ff check default user policy 2024-10-23 15:29:47 +04:00
abhishek9686
c0f107b302 remove userips usage, add allow all to fwupdate 2024-10-23 14:15:13 +04:00