Commit graph

1224 commits

Author SHA1 Message Date
Abhishek K
5849e0e2e1
NET-1932: sort out acls into CE and PRO (#3460)
* move relevant acl and tag code to CE and Pro pkgs

* intialise pro acl funcs

* list gateways by user access

* check user gw access by policies

* filter out user policies on CE

* filter out tagged policies on CE

* fix ce acl comms

* allow gateways tag

* allow gateway tag  on CE, remove failover and gw check on acl policy

* add gw rules func to pro

* add inet gw support on CE

* add egress acl API

* add egress acl API

* fix(go): set is_gw when converting api node to server node;

* fix(go): set is_gw when converting api node to server node;

* fix policy validity checker for inet gws

* move dns option to host model

* fix node removal from egress policy on delete

* add migration logic for ManageDNS

* fix dns json field

* fix nil error on node tags

* add egress info to relayed nodes

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-06-02 13:05:10 +05:30
abhishek9686
84573787f8 add user info to events 2025-05-27 23:00:18 +05:30
abhishek9686
f538e80f76 fix enrollment key join on existing networks 2025-05-27 22:55:16 +05:30
Abhishek K
a1304b43d8
NET-2054: Auto Removal of Offline Nodes, fix enrollment key relay function (#3458)
* check host ports on join

* if 443 not available fallback to 51821

* if 443 not available fallback to 51821

* add config for auto delete of offline nodes

* autocleanup offline nodes

* delete offline nodes on startup

* fix relay via join token
2025-05-24 08:21:47 +05:30
abhishek9686
5ade7e9557 add sync,upragde,refresh keys events 2025-05-22 15:47:57 +05:30
abhishek9686
771d941868 copy user diff before update 2025-05-22 13:59:54 +05:30
Vishal Dalwadi
614cf77b5a
NET-1991: Add IDP sync functionality. (#3428)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* feat(go): add types for idp package;

* feat(go): import azure sdk;

* feat(go): add stub for google workspace client;

* feat(go): implement azure ad client;

* feat(go): sync users and groups using idp client;

* publish peer update on settings update

* feat(go): read creds from env vars;

* feat(go): add api endpoint to trigger idp sync;

* fix(go): sync member changes;

* fix(go): handle error;

* fix(go): set correct response type;

* feat(go): support disabling user accounts;

1. Add api endpoints to enable and disable user accounts.
2. Add checks in authenticators to prevent disabled users from logging in.
3. Add checks in middleware to prevent api usage by disabled users.

* feat(go): use string slice for group members;

* feat(go): sync user account status from idp;

* feat(go): import google admin sdk;

* feat(go): add support for google workspace idp;

* feat(go): initialize idp client on sync;

* feat(go): sync from idp periodically;

* feat(go): improvements for google idp;

1. Use the impersonate package to authenticate.
2. Use Pages method to get all data.

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* feat(go): fetch idp creds from server settings;

* feat(go): add filters for users and groups;

* feat(go): skip sync from idp if disabled;

* feat(go): add endpoint to remove idp integration;

* feat(go): import all users if no filters;

* feat(go): assign service-user role on sync;

* feat(go): remove microsoft-go-sdk;

* feat(go): add display name field for user;

* fix(go): set account disabled correctly;

* fix(go): update user if display name changes;

* fix(go): remove auth provider when removing idp integration;

* fix(go): ignore display name if empty;

* feat(go): add idp sync interval setting;

* fix(go): error on invalid auth provider;

* fix(go): no error if no user on group delete;

* fix(go): check superadmin using platform role id;

* feat(go): add display name and account disabled to return user as well;

* feat(go): tidy go mod after merge;

* feat(go): reinitialize auth provider and idp sync hook;

* fix(go): merge error;

* fix(go): merge error;

* feat(go): use id as the external provider id;

* fix(go): comments;

* feat(go): add function to return pending users;

* feat(go): prevent external id erasure;

* fix(go): user and group sync errors;

* chore(go): cleanup;

* fix(go): delete only oauth users;

* feat(go): use uuid group id;

* export ipd id to in rest api

* feat(go): don't use uuid for default groups;

* feat(go): migrate group only if id not uuid;

* chore(go): go mod tidy;

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
Co-authored-by: Abhishek K <abhishek@netmaker.io>
Co-authored-by: the_aceix <aceixsmartx@gmail.com>
2025-05-21 13:48:15 +05:30
Abhishek K
d7bad9865a
NET-2014: Audit Logging (#3455)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* egress model

* fix revoked tokens to be unauthorized

* update egress model

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* setup api handlers for egress revamp

* use single DB, fix update nat boolean field

* extend validaiton checks for egress ranges

* add migration to convert to new egress model

* fix panic interface conversion

* publish peer update on settings update

* revoke token generated by an user

* add user token creation restriction by user role

* add forbidden check for access token creation

* revoke user token when group or role is changed

* add default group to admin users on update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* remove nat check on egress gateway request

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* add inet gw status to egress model

* fetch node ids in the tag, add inet gw info clients

* add inet gw info to node from egress list

* add migration logic internet gws

* create default acl policies

* add egress info

* add egress TODO

* add egress TODO

* fix user auth api:

* add reference id to acl policy

* add egress response from DB

* publish peer update on egress changes

* re initalise oauth and email config

* set verbosity

* normalise cidr on egress req

* add egress id to acl group

* change acls to use egress id

* resolve merge conflicts

* fix egress reference errors

* move egress model to schema

* add api context to DB

* sync auto update settings with hosts

* sync auto update settings with hosts

* check acl for egress node

* check for egress policy in the acl dst groups

* fix acl rules for egress policies with new models

* add status to egress model

* fix inet node func

* mask secret and convert jwt duration to minutes

* enable egress policies on creation

* convert jwt duration to minutes

* add relevant ranges to inet egress

* skip non active egress routes

* resolve merge conflicts

* fix static check

* notify peers after settings update

* define schema for activity, add api handler to list network activity

* setup event channel and logger

* setup event logger, add event for user login

* change activity model to event

* add api error constants

* add logout event

* log user crud events

* add login events for oauth

* add user related events

* log events for invites and user approvals

* order user activity event by timestamp

* fix logout api

* add user and network events api, add addtional events triggers

* add filters to all events api

* fix events filter

* add diff to event logs

* update user logout api

* log settigns updates

* log events for network and host updates

* check for diff on events

* log host del event

* add user loc info to desktop app connection events

* fix authorize middleware check

* add gateway events

* resolve merge conflicts

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-05-21 13:13:20 +05:30
Abhishek K
307a3d1e4b
NET-1932: Merge egress and internet gateways (#3436)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* egress model

* fix revoked tokens to be unauthorized

* update egress model

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* setup api handlers for egress revamp

* use single DB, fix update nat boolean field

* extend validaiton checks for egress ranges

* add migration to convert to new egress model

* fix panic interface conversion

* publish peer update on settings update

* revoke token generated by an user

* add user token creation restriction by user role

* add forbidden check for access token creation

* revoke user token when group or role is changed

* add default group to admin users on update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* remove nat check on egress gateway request

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* add inet gw status to egress model

* fetch node ids in the tag, add inet gw info clients

* add inet gw info to node from egress list

* add migration logic internet gws

* create default acl policies

* add egress info

* add egress TODO

* add egress TODO

* fix user auth api:

* add reference id to acl policy

* add egress response from DB

* publish peer update on egress changes

* re initalise oauth and email config

* set verbosity

* normalise cidr on egress req

* add egress id to acl group

* change acls to use egress id

* resolve merge conflicts

* fix egress reference errors

* move egress model to schema

* add api context to DB

* sync auto update settings with hosts

* sync auto update settings with hosts

* check acl for egress node

* check for egress policy in the acl dst groups

* fix acl rules for egress policies with new models

* add status to egress model

* fix inet node func

* mask secret and convert jwt duration to minutes

* enable egress policies on creation

* convert jwt duration to minutes

* add relevant ranges to inet egress

* skip non active egress routes

* resolve merge conflicts

* fix static check

* update gorm tag for primary key on egress model

* create user policies for egress resources

* resolve merge conflicts

* get egress info on failover apis, add egress src validation for inet gws

* add additional validation checks on egress req

* add additional validation checks on egress req

* skip all resources for inet policy

* delete associated egress acl policies

* fix failover of inetclient

* avoid setting inet client asd inet gw

* fix all resource egress policy

* fix inet gw egress rule

* check for node egress on relay req

* fix egress acl rules comms

* add new field for egress info on node

* check acl policy in failover ctx

* avoid default host to be set as inet client

* fix relayed egress node

* add valid error messaging for egress validate func

* return if inet default host

* jump port detection to 51821

* check host ports on pull

* check user access gws via acls

* add validation check for default host and failover for inet clients

* add error messaging for acl policy check

* fix inet gw status

* ignore failover req for peer using inet gw

* check for allowed egress ranges for a peer

* add egress routes to static nodes by access

* avoid setting failvoer as inet client

* fix egress error messaging

* fix extclients egress comms

* fix inet gw acting as inet client

* return formatted error on update acl validation

* add default route for static nodes on inetclient

* check relay node acting as inetclient

* move inet node info to separate field, fix all resouces policy

* remove debug logs

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-05-21 12:50:21 +05:30
Aceix
506f73ebb9
Merge pull request #3441 from gravitl/nmctl-access-token
feat: add support for user access tokens in nmctl
2025-05-20 09:16:24 +05:30
Abhishek K
f9bc3a5386
Merge pull request #3440 from gravitl/master
Master
2025-04-30 03:02:05 +04:00
Abhishek K
85084f9dfe
check host ports on join (#3437) 2025-04-30 03:00:36 +04:00
Aceix
b0a8a07ebd
fix: add node status in response to create/delete gateway (#3425) 2025-04-30 02:57:53 +04:00
alingse
eb0dbf7ee4
fix call logic.FormatError with a nil value error (#3406)
* fix call logic.FormatError with a nil value error

Signed-off-by: alingse <alingse@foxmail.com>

* fix call slog.Error with wrong error

---------

Signed-off-by: alingse <alingse@foxmail.com>
2025-04-30 02:39:35 +04:00
Aceix
c9f3750499
fix: improve error response message (#3426) 2025-04-30 02:34:45 +04:00
Abhishek K
309e4795a1
NET-1950: Persist Server Settings in the DB (#3419)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* publish peer update on settings update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* set verbosity

* sync auto update settings with hosts

* sync auto update settings with hosts

* mask secret and convert jwt duration to minutes

* convert jwt duration to minutes

* notify peers after settings update

* compare with curr settings before updating

* send host update to devices on auto update

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-04-30 02:34:10 +04:00
Abhishek K
262803c234
add ctx to DB funcs (#3435) 2025-04-29 00:22:02 +04:00
Abhishek K
119ef4e17e
move api tokens models to schema pkg (#3434) 2025-04-29 00:11:56 +04:00
Abhishek K
ca95954fb5
NET-2000: Api access tokens (#3418)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* revoke token generated by an user

* add user token creation restriction by user role

* add forbidden check for access token creation

* revoke user token when group or role is changed

* add default group to admin users on update

* fix token removal on user update

* fix token removal on user update
2025-04-23 20:21:42 +04:00
abhishek9686
b5842b7b06 fix create gateway req 2025-04-03 21:47:41 +04:00
abhishek9686
3bb843e96e fix openapi tag 2025-04-02 19:18:41 +04:00
Abhishek K
4c26ae5197
Merge pull request #3391 from gravitl/master
Master
2025-03-25 11:51:06 +04:00
Abhishek K
8a1f36ee8e
Net 1990 (#3386)
* add peerkey to network egress routes model

* add peerkey to network egress routes model

* filter out conflicting routes from node

* add support for egress HA on relay

* add support for egress HA on relay

* add support for egress HA on relay

* skip if curr node is relay node of the peer

* skip if curr node is relay node of the peer

* fix failover egress HA

* add network to egress route model

* clone before modifying

* check for user policies on uni direction check

* add all network fwd rule

* delete all network policies and tags on network deletion
2025-03-25 00:42:14 +04:00
Abhishek K
5e1ef5e366
Net 1990 (#3385)
* add peerkey to network egress routes model

* add peerkey to network egress routes model

* filter out conflicting routes from node

* add support for egress HA on relay

* add support for egress HA on relay

* add support for egress HA on relay

* skip if curr node is relay node of the peer

* skip if curr node is relay node of the peer

* fix failover egress HA

* add network to egress route model

* clone before modifying

* check for user policies on uni direction check

* add all network fwd rule
2025-03-25 00:29:35 +04:00
Abhishek K
3d765f9cf1
NET-1910: Acl controls for Egress Traffic (#3377)
* add support for egress ranges on acl policy

* add egress ranges to acl rules

* add egress ranges to acl policies

* Add egress ranges to acl rules

* add egress ranges to fw update

* fetch acl rules for egress networks

* apply egress policies for devices

* configure user policies for egresss routes

* fix gw tag name migration

* fix egress acl rules for static nodes

* add egress ranges for static nodes on ingress gw

* fileter acl IPs to be unique

* cleanup IOT logic from peer update

* make acl Rule Dst List

* cleanup egress ranges from acl policies

* create user group default acl policy for gateways

* remove remote access name ids

* rm egress ranges removal from acl policies

* simplify user permissions on nodes

* add additional nameservers to extclient dns

* remove debug logs

* fix static checks
2025-03-18 13:25:55 +04:00
Aceix
372c797e60
fix: add status to updated node response during update (#3363) 2025-03-18 09:34:42 +04:00
Vishal Dalwadi
4c14cfd099
NET-1986: Only report online hosts and external clients (#3375)
* feat(go): only report online hosts.

* feat(go): only report online external clients.

* feat(go): only report online hosts and external clients.
2025-03-17 18:35:17 +04:00
Aceix
7c262abbb9
Merge pull request #3356 from gravitl/fix/allow-multiple-client-gw-dns
Fix/allow-multiple-client-dns
2025-03-11 00:53:38 +04:00
abhishek9686
bb25739bd9 resolve merge conflicts 2025-03-06 22:41:56 +04:00
Abhishek K
1ad8b8b7b4
Merge pull request #3358 from gravitl/master
Master
2025-03-06 22:17:24 +04:00
abhishek9686
b45a926649 optimise static node rules, fix traffic flows for static nodes 2025-03-05 14:26:35 +04:00
abhishek9686
d21411392b fix integration tests 2025-03-03 12:05:23 +04:00
abhishek9686
48b805bbc2 add loadbalance endpoint for extclients 2025-03-02 20:59:53 +04:00
abhishek9686
33fafa83e6 add nameservers to dns settins 2025-02-28 21:49:53 +04:00
abhishek9686
21826d41b6 add nameservers to network settings 2025-02-28 21:27:18 +04:00
abhishek9686
225bf37e02 resolve merge conflicts 2025-02-28 12:18:38 +04:00
abhishek9686
c0a88e7c43 add cache internal to the func 2025-02-28 11:09:39 +04:00
abhishek9686
ce6ec25253 fix network ip allocation in HA 2025-02-27 16:26:15 +04:00
abhishek9686
e50da805d4 Merge branch 'develop' of https://github.com/gravitl/netmaker into NET-1911 2025-02-24 08:53:49 +03:00
Abhishek K
48535f7ef1
NET-1956: Async Node Status API (#3341)
* add node status api

* upsate node status api to return map data

* resolve merge conflicts
2025-02-24 08:48:24 +03:00
Aceix
2fcdd865c8
fix: unlnk deleted networks and tags from enrollment keys (#3328) 2025-02-24 08:22:18 +03:00
Aceix
f46030e40f
fix: add user group struct to get user response (#3336)
* fix: add user group struct to user response

* fix: set auto-created network groups as default
2025-02-24 08:17:55 +03:00
abhishek9686
9590f9e5a4 resolve merge conflicts 2025-02-19 09:33:23 +04:00
Abhishek K
ee6bea117f
Merge branch 'develop' into master 2025-02-19 01:14:06 +04:00
abhishek9686
7dd4c048c3 add mutex around peer tags 2025-02-18 22:10:03 +04:00
abhishek9686
41fa0b1bce resolve merge conflicts 2025-02-14 18:24:56 +04:00
Abhishek K
689b9b6b52
NET-1941: failover signaling optimized, new endpoint for metrics peer info (#3322)
* optimise acl cache with network map

* fix mutex clash

* fix acl mutex

* add new endpoint for metrics peer info

* fix static check

* add new endpoint for metrics peer info

* fix host peer info api params

* cache failover data

* add json tag

* avoid duplicate calls to failover

* add failover check ctx

* add failover check ctx

* fix failover debug log

* optimise failover operations

* remove debug logs

* rm unused model

* rm unused model
2025-02-14 16:06:26 +04:00
abhishek9686
61bf479e50 upsert on color update 2025-02-07 17:23:53 +04:00
abhishek9686
9cf2ad4fc5 add color coding to tags 2025-02-06 15:14:15 +04:00
abhishek9686
a9c14c4643 Merge branch 'develop' of https://github.com/gravitl/netmaker into NET-1911 2025-02-04 19:32:25 +04:00