Commit graph

77 commits

Author SHA1 Message Date
abhishek9686
885b36810b depreacte ingress dns usage on client configs 2025-08-27 14:04:57 +05:30
abhishek9686
b043ed8cb7 add migration func for old upstream servers 2025-08-26 23:53:09 +05:30
abhishek9686
18f74e5539 update match domain to list 2025-08-26 18:11:55 +05:30
Abhishek K
e517a0ac66
fix egress HA migration (#3555) 2025-07-18 12:33:21 +05:30
abhishek9686
4e2b7993eb fix merge conflicts 2025-06-24 17:00:43 +05:30
abhishek9686
f44c37e3aa fix merge conflicts 2025-06-24 17:00:10 +05:30
abhishek9686
b423b183e4 fix merge conflicts 2025-06-24 16:56:48 +05:30
Abhishek K
0a47cc5461
Merge branch 'develop' into NET-1994 2025-06-23 17:50:21 +05:30
abhishek9686
2f5acf6742 sync changes on startup, add create relay calls on defaul host 2025-06-23 06:28:43 +05:30
abhishek9686
d978de08d0 collect host localtion for graph 2025-06-12 15:47:24 +05:30
abhishek9686
cffc1d35ff reduce log retention rate to 7 days 2025-06-10 22:36:40 +05:30
abhishek9686
087b7393d6 update dns settigns on migration 2025-06-09 17:49:10 +05:30
abhishek9686
6be6304a64 fix host dns setting 2025-06-09 17:17:26 +05:30
Abhishek K
810ff21165
NET-2014: add audit log retention period, add timestamp for events (#3486)
* revert inet gws from acl policies

* add egress range with metric for inet gw

* link pro inet funcs

* add timestamp params to activity apis
2025-06-06 13:19:56 +05:30
Abhishek K
0f884d4f36
NET-2061: revert Inet gws, fix extclient comms with user policies (#3482)
* revert inet gws from acl policies

* add egress range with metric for inet gw

* link pro inet funcs

* fix extclient comms with users

* remove TODO comments
2025-06-05 22:42:16 +05:30
Abhishek K
c1304aea5e
NET-1932: Migration Fix (#3470)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

* fix default network user policy

* fix egress migration

* fix egress migration

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-03 08:24:05 +05:30
Abhishek K
5849e0e2e1
NET-1932: sort out acls into CE and PRO (#3460)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-02 13:05:10 +05:30
Vishal Dalwadi
f0affca56b
NET-1991 Fixes (#3467)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* feat(go): add types for idp package;

* feat(go): import azure sdk;

* feat(go): add stub for google workspace client;

* feat(go): implement azure ad client;

* feat(go): sync users and groups using idp client;

* publish peer update on settings update

* feat(go): read creds from env vars;

* feat(go): add api endpoint to trigger idp sync;

* fix(go): sync member changes;

* fix(go): handle error;

* fix(go): set correct response type;

* feat(go): support disabling user accounts;

1. Add api endpoints to enable and disable user accounts.
2. Add checks in authenticators to prevent disabled users from logging in.
3. Add checks in middleware to prevent api usage by disabled users.

* feat(go): use string slice for group members;

* feat(go): sync user account status from idp;

* feat(go): import google admin sdk;

* feat(go): add support for google workspace idp;

* feat(go): initialize idp client on sync;

* feat(go): sync from idp periodically;

* feat(go): improvements for google idp;

1. Use the impersonate package to authenticate.
2. Use Pages method to get all data.

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* feat(go): fetch idp creds from server settings;

* feat(go): add filters for users and groups;

* feat(go): skip sync from idp if disabled;

* feat(go): add endpoint to remove idp integration;

* feat(go): import all users if no filters;

* feat(go): assign service-user role on sync;

* feat(go): remove microsoft-go-sdk;

* feat(go): add display name field for user;

* fix(go): set account disabled correctly;

* fix(go): update user if display name changes;

* fix(go): remove auth provider when removing idp integration;

* fix(go): ignore display name if empty;

* feat(go): add idp sync interval setting;

* fix(go): error on invalid auth provider;

* fix(go): no error if no user on group delete;

* fix(go): check superadmin using platform role id;

* feat(go): add display name and account disabled to return user as well;

* feat(go): tidy go mod after merge;

* feat(go): reinitialize auth provider and idp sync hook;

* fix(go): merge error;

* fix(go): merge error;

* feat(go): use id as the external provider id;

* fix(go): comments;

* feat(go): add function to return pending users;

* feat(go): prevent external id erasure;

* fix(go): user and group sync errors;

* chore(go): cleanup;

* fix(go): delete only oauth users;

* feat(go): use uuid group id;

* export ipd id to in rest api

* feat(go): don't use uuid for default groups;

* feat(go): migrate group only if id not uuid;

* chore(go): go mod tidy;

* fix(go): empty id user groups membership;

* fix(go): set user groups only if nil;

* fix(go): use reset to start only if required;

* fix(go): use context for better control;

* fix(go): allow id to be string or int;

* fix(go): roles and groups migration;

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
Co-authored-by: Abhishek K <abhishek@netmaker.io>
Co-authored-by: the_aceix <aceixsmartx@gmail.com>
2025-05-29 12:32:12 +05:30
Vishal Dalwadi
614cf77b5a
NET-1991: Add IDP sync functionality. (#3428)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* feat(go): add types for idp package;

* feat(go): import azure sdk;

* feat(go): add stub for google workspace client;

* feat(go): implement azure ad client;

* feat(go): sync users and groups using idp client;

* publish peer update on settings update

* feat(go): read creds from env vars;

* feat(go): add api endpoint to trigger idp sync;

* fix(go): sync member changes;

* fix(go): handle error;

* fix(go): set correct response type;

* feat(go): support disabling user accounts;

1. Add api endpoints to enable and disable user accounts.
2. Add checks in authenticators to prevent disabled users from logging in.
3. Add checks in middleware to prevent api usage by disabled users.

* feat(go): use string slice for group members;

* feat(go): sync user account status from idp;

* feat(go): import google admin sdk;

* feat(go): add support for google workspace idp;

* feat(go): initialize idp client on sync;

* feat(go): sync from idp periodically;

* feat(go): improvements for google idp;

1. Use the impersonate package to authenticate.
2. Use Pages method to get all data.

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* feat(go): fetch idp creds from server settings;

* feat(go): add filters for users and groups;

* feat(go): skip sync from idp if disabled;

* feat(go): add endpoint to remove idp integration;

* feat(go): import all users if no filters;

* feat(go): assign service-user role on sync;

* feat(go): remove microsoft-go-sdk;

* feat(go): add display name field for user;

* fix(go): set account disabled correctly;

* fix(go): update user if display name changes;

* fix(go): remove auth provider when removing idp integration;

* fix(go): ignore display name if empty;

* feat(go): add idp sync interval setting;

* fix(go): error on invalid auth provider;

* fix(go): no error if no user on group delete;

* fix(go): check superadmin using platform role id;

* feat(go): add display name and account disabled to return user as well;

* feat(go): tidy go mod after merge;

* feat(go): reinitialize auth provider and idp sync hook;

* fix(go): merge error;

* fix(go): merge error;

* feat(go): use id as the external provider id;

* fix(go): comments;

* feat(go): add function to return pending users;

* feat(go): prevent external id erasure;

* fix(go): user and group sync errors;

* chore(go): cleanup;

* fix(go): delete only oauth users;

* feat(go): use uuid group id;

* export ipd id to in rest api

* feat(go): don't use uuid for default groups;

* feat(go): migrate group only if id not uuid;

* chore(go): go mod tidy;

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
Co-authored-by: Abhishek K <abhishek@netmaker.io>
Co-authored-by: the_aceix <aceixsmartx@gmail.com>
2025-05-21 13:48:15 +05:30
Abhishek K
307a3d1e4b
NET-1932: Merge egress and internet gateways (#3436)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* egress model

* fix revoked tokens to be unauthorized

* update egress model

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* setup api handlers for egress revamp

* use single DB, fix update nat boolean field

* extend validaiton checks for egress ranges

* add migration to convert to new egress model

* fix panic interface conversion

* publish peer update on settings update

* revoke token generated by an user

* add user token creation restriction by user role

* add forbidden check for access token creation

* revoke user token when group or role is changed

* add default group to admin users on update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* remove nat check on egress gateway request

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* add inet gw status to egress model

* fetch node ids in the tag, add inet gw info clients

* add inet gw info to node from egress list

* add migration logic internet gws

* create default acl policies

* add egress info

* add egress TODO

* add egress TODO

* fix user auth api:

* add reference id to acl policy

* add egress response from DB

* publish peer update on egress changes

* re initalise oauth and email config

* set verbosity

* normalise cidr on egress req

* add egress id to acl group

* change acls to use egress id

* resolve merge conflicts

* fix egress reference errors

* move egress model to schema

* add api context to DB

* sync auto update settings with hosts

* sync auto update settings with hosts

* check acl for egress node

* check for egress policy in the acl dst groups

* fix acl rules for egress policies with new models

* add status to egress model

* fix inet node func

* mask secret and convert jwt duration to minutes

* enable egress policies on creation

* convert jwt duration to minutes

* add relevant ranges to inet egress

* skip non active egress routes

* resolve merge conflicts

* fix static check

* update gorm tag for primary key on egress model

* create user policies for egress resources

* resolve merge conflicts

* get egress info on failover apis, add egress src validation for inet gws

* add additional validation checks on egress req

* add additional validation checks on egress req

* skip all resources for inet policy

* delete associated egress acl policies

* fix failover of inetclient

* avoid setting inet client asd inet gw

* fix all resource egress policy

* fix inet gw egress rule

* check for node egress on relay req

* fix egress acl rules comms

* add new field for egress info on node

* check acl policy in failover ctx

* avoid default host to be set as inet client

* fix relayed egress node

* add valid error messaging for egress validate func

* return if inet default host

* jump port detection to 51821

* check host ports on pull

* check user access gws via acls

* add validation check for default host and failover for inet clients

* add error messaging for acl policy check

* fix inet gw status

* ignore failover req for peer using inet gw

* check for allowed egress ranges for a peer

* add egress routes to static nodes by access

* avoid setting failvoer as inet client

* fix egress error messaging

* fix extclients egress comms

* fix inet gw acting as inet client

* return formatted error on update acl validation

* add default route for static nodes on inetclient

* check relay node acting as inetclient

* move inet node info to separate field, fix all resouces policy

* remove debug logs

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-05-21 12:50:21 +05:30
Abhishek K
309e4795a1
NET-1950: Persist Server Settings in the DB (#3419)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* publish peer update on settings update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* set verbosity

* sync auto update settings with hosts

* sync auto update settings with hosts

* mask secret and convert jwt duration to minutes

* convert jwt duration to minutes

* notify peers after settings update

* compare with curr settings before updating

* send host update to devices on auto update

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-04-30 02:34:10 +04:00
Abhishek K
ca95954fb5
NET-2000: Api access tokens (#3418)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* revoke token generated by an user

* add user token creation restriction by user role

* add forbidden check for access token creation

* revoke user token when group or role is changed

* add default group to admin users on update

* fix token removal on user update

* fix token removal on user update
2025-04-23 20:21:42 +04:00
Vishal Dalwadi
7f40371ffd
Task/DB-Migration: Add Key-Value to SQL Migration functionality. (#3380)
* feat(go): add db package;

* feat(go): add jobs table;

* feat(go): add schema migration facade;

* refactor(go): use custom key type to avoid collisions;
2025-04-12 14:07:57 +04:00
Abhishek K
9b2bffe723
Merge pull request #3396 from gravitl/release-v0.30.0
add migration for extclient tags
2025-03-25 17:02:34 +04:00
abhishek9686
75307cb726 add migration for extclient tags 2025-03-25 17:01:03 +04:00
Abhishek K
3d765f9cf1
NET-1910: Acl controls for Egress Traffic (#3377)
* add support for egress ranges on acl policy

* add egress ranges to acl rules

* add egress ranges to acl policies

* Add egress ranges to acl rules

* add egress ranges to fw update

* fetch acl rules for egress networks

* apply egress policies for devices

* configure user policies for egresss routes

* fix gw tag name migration

* fix egress acl rules for static nodes

* add egress ranges for static nodes on ingress gw

* fileter acl IPs to be unique

* cleanup IOT logic from peer update

* make acl Rule Dst List

* cleanup egress ranges from acl policies

* create user group default acl policy for gateways

* remove remote access name ids

* rm egress ranges removal from acl policies

* simplify user permissions on nodes

* add additional nameservers to extclient dns

* remove debug logs

* fix static checks
2025-03-18 13:25:55 +04:00
abhishek9686
f3fd10326f migrate remote access gw tags 2025-03-11 00:50:44 +04:00
abhishek9686
d9f58efd05 add migration func for egress 2025-03-03 00:56:16 +04:00
Abhishek K
8297642b90
NET-1914: add gw apis, move relays into CE (#3309)
* add gw apis, move relays into CE

* set gw field on relay and ingress creation

* add gw handlers to relay and ingress apis

* if node is inetgw and gw add dns

* remove pro check on relays

* fetch node before updating
2025-01-28 11:28:31 +05:30
Abhishek K
7093373c77
NET-1784: Migrate All Policies (#3245)
* migrate all policies

* migrate all policies

* add acl policy type for any traffic

* use any service type for migration
2024-12-10 11:55:37 +04:00
Abhishek K
f124b10c35
NET-1784: Adv Acl Rules (#3239)
* define direction on acl req

* define protocol types and rule model

* get rules for node

* fetch acl rule for a node

* redine acl firewall model

* add json tags

* update port,protocol, and direction

* add json tags to acl options

* convert protocol to string

* simplify acl map

* add json tags to acl rules

* add networks to fw update

* add acls rules

* NET-1784: add allow all field

* add allow all field on fw udpate

* remove debug logs

* fix port and protocol types

* migrate default acl policies

* define constants for service types

* add adv options for user rules on ingress gw

* debug log

* allow whole network

* add static nodes to acl rules

* replace peers on acl updates

* initiliase rule map

* add user acl rules on target node

* revert acl check on extclient

* handle static node rules on ingress gw

* update multiple policies for users

* check allowed direction

* remove debug logs
2024-12-10 11:21:14 +04:00
Max Ma
e8d3e80b08 initialize cache in startup 2024-11-29 11:35:34 +01:00
abhishek9686
df7a1fae65 sync hosts into cache 2024-11-27 15:30:20 +04:00
abhishek9686
cbe660fbf2 get all nodes on startup 2024-11-27 15:01:26 +04:00
abhishek9686
dcbe94eeb5 avoid adding static nodes to tags 2024-11-11 21:31:10 +04:00
abhishek9686
ce93c9e0c1 add migrate fix on tags init 2024-11-08 10:55:43 +04:00
abhishek9686
42b8a03660 fix user migration 2024-10-31 12:06:14 +04:00
abhishek9686
4b99435cbc user roles and groups migration 2024-10-31 11:58:54 +04:00
abhishek9686
167d29a96b remove * on default grp, add admins to admin net groups by default 2024-10-30 19:22:05 +04:00
abhishek9686
56d5c85da7 block default key deletion,delete default key on network deletion 2024-10-30 15:58:55 +04:00
abhishek9686
0ddbf7958e fix rebase errors 2024-10-30 12:02:42 +04:00
abhishek9686
0d6792c21b Merge branch 'develop' of https://github.com/gravitl/netmaker into NET-1615 2024-10-30 11:49:54 +04:00
Abhishek K
f63ed23b61
Merge pull request #3160 from gravitl/NET-1640
NET-1640: Include static Nodes in the nodes api
2024-10-30 11:49:10 +04:00
abhishek9686
4eacadbb27 convert network roles to groups, deprecate RAG roles 2024-10-28 16:18:50 +04:00
abhishek9686
60ea2a76a6 create enrollementkey for existing networks 2024-10-24 13:11:53 +04:00
abhishek9686
c37cf2b7e3 create default user policies 2024-10-18 16:53:39 +04:00
abhishek9686
afbd725864 Merge branch 'NET-1613' of https://github.com/gravitl/netmaker into NET-1615-ext 2024-10-18 14:26:31 +04:00
abhishek9686
d8a0398b2a add remoteaccess tag to extclients 2024-10-18 14:25:55 +04:00
abhishek9686
725cb569dc Merge branch 'NET-1613' of https://github.com/gravitl/netmaker into NET-1615 2024-10-17 11:31:21 +04:00
abhishek9686
14152a9484 add remote access tags in migration 2024-10-17 11:30:28 +04:00