Commit graph

75 commits

Author SHA1 Message Date
Vishal Dalwadi
614cf77b5a
NET-1991: Add IDP sync functionality. (#3428)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* feat(go): add types for idp package;

* feat(go): import azure sdk;

* feat(go): add stub for google workspace client;

* feat(go): implement azure ad client;

* feat(go): sync users and groups using idp client;

* publish peer update on settings update

* feat(go): read creds from env vars;

* feat(go): add api endpoint to trigger idp sync;

* fix(go): sync member changes;

* fix(go): handle error;

* fix(go): set correct response type;

* feat(go): support disabling user accounts;

1. Add api endpoints to enable and disable user accounts.
2. Add checks in authenticators to prevent disabled users from logging in.
3. Add checks in middleware to prevent api usage by disabled users.

* feat(go): use string slice for group members;

* feat(go): sync user account status from idp;

* feat(go): import google admin sdk;

* feat(go): add support for google workspace idp;

* feat(go): initialize idp client on sync;

* feat(go): sync from idp periodically;

* feat(go): improvements for google idp;

1. Use the impersonate package to authenticate.
2. Use Pages method to get all data.

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* feat(go): fetch idp creds from server settings;

* feat(go): add filters for users and groups;

* feat(go): skip sync from idp if disabled;

* feat(go): add endpoint to remove idp integration;

* feat(go): import all users if no filters;

* feat(go): assign service-user role on sync;

* feat(go): remove microsoft-go-sdk;

* feat(go): add display name field for user;

* fix(go): set account disabled correctly;

* fix(go): update user if display name changes;

* fix(go): remove auth provider when removing idp integration;

* fix(go): ignore display name if empty;

* feat(go): add idp sync interval setting;

* fix(go): error on invalid auth provider;

* fix(go): no error if no user on group delete;

* fix(go): check superadmin using platform role id;

* feat(go): add display name and account disabled to return user as well;

* feat(go): tidy go mod after merge;

* feat(go): reinitialize auth provider and idp sync hook;

* fix(go): merge error;

* fix(go): merge error;

* feat(go): use id as the external provider id;

* fix(go): comments;

* feat(go): add function to return pending users;

* feat(go): prevent external id erasure;

* fix(go): user and group sync errors;

* chore(go): cleanup;

* fix(go): delete only oauth users;

* feat(go): use uuid group id;

* export ipd id to in rest api

* feat(go): don't use uuid for default groups;

* feat(go): migrate group only if id not uuid;

* chore(go): go mod tidy;

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
Co-authored-by: Abhishek K <abhishek@netmaker.io>
Co-authored-by: the_aceix <aceixsmartx@gmail.com>
2025-05-21 13:48:15 +05:30
Abhishek K
d7bad9865a
NET-2014: Audit Logging (#3455)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* egress model

* fix revoked tokens to be unauthorized

* update egress model

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* setup api handlers for egress revamp

* use single DB, fix update nat boolean field

* extend validaiton checks for egress ranges

* add migration to convert to new egress model

* fix panic interface conversion

* publish peer update on settings update

* revoke token generated by an user

* add user token creation restriction by user role

* add forbidden check for access token creation

* revoke user token when group or role is changed

* add default group to admin users on update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* remove nat check on egress gateway request

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* add inet gw status to egress model

* fetch node ids in the tag, add inet gw info clients

* add inet gw info to node from egress list

* add migration logic internet gws

* create default acl policies

* add egress info

* add egress TODO

* add egress TODO

* fix user auth api:

* add reference id to acl policy

* add egress response from DB

* publish peer update on egress changes

* re initalise oauth and email config

* set verbosity

* normalise cidr on egress req

* add egress id to acl group

* change acls to use egress id

* resolve merge conflicts

* fix egress reference errors

* move egress model to schema

* add api context to DB

* sync auto update settings with hosts

* sync auto update settings with hosts

* check acl for egress node

* check for egress policy in the acl dst groups

* fix acl rules for egress policies with new models

* add status to egress model

* fix inet node func

* mask secret and convert jwt duration to minutes

* enable egress policies on creation

* convert jwt duration to minutes

* add relevant ranges to inet egress

* skip non active egress routes

* resolve merge conflicts

* fix static check

* notify peers after settings update

* define schema for activity, add api handler to list network activity

* setup event channel and logger

* setup event logger, add event for user login

* change activity model to event

* add api error constants

* add logout event

* log user crud events

* add login events for oauth

* add user related events

* log events for invites and user approvals

* order user activity event by timestamp

* fix logout api

* add user and network events api, add addtional events triggers

* add filters to all events api

* fix events filter

* add diff to event logs

* update user logout api

* log settigns updates

* log events for network and host updates

* check for diff on events

* log host del event

* add user loc info to desktop app connection events

* fix authorize middleware check

* add gateway events

* resolve merge conflicts

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-05-21 13:13:20 +05:30
Abhishek K
309e4795a1
NET-1950: Persist Server Settings in the DB (#3419)
* feat: api access tokens

* revoke all user tokens

* redefine access token api routes, add auto egress option to enrollment keys

* add server settings apis, add db table for settigs

* handle server settings updates

* switch to using settings from DB

* fix sever settings migration

* revet force migration for settings

* fix server settings database write

* fix revoked tokens to be unauthorized

* remove unused functions

* convert access token to sql schema

* switch access token to sql schema

* fix merge conflicts

* fix server settings types

* bypass basic auth setting for super admin

* add TODO comment

* publish peer update on settings update

* chore(go): import style changes from migration branch;

1. Singular file names for table schema.
2. No table name method.
3. Use .Model instead of .Table.
4. No unnecessary tagging.

* remove nat check on egress gateway request

* Revert "remove nat check on egress gateway request"

This reverts commit 0aff12a189.

* feat(go): add db middleware;

* feat(go): restore method;

* feat(go): add user access token schema;

* fix user auth api:

* re initalise oauth and email config

* set verbosity

* sync auto update settings with hosts

* sync auto update settings with hosts

* mask secret and convert jwt duration to minutes

* convert jwt duration to minutes

* notify peers after settings update

* compare with curr settings before updating

* send host update to devices on auto update

---------

Co-authored-by: Vishal Dalwadi <dalwadivishal26@gmail.com>
2025-04-30 02:34:10 +04:00
Vishal Dalwadi
4c14cfd099
NET-1986: Only report online hosts and external clients (#3375)
* feat(go): only report online hosts.

* feat(go): only report online external clients.

* feat(go): only report online hosts and external clients.
2025-03-17 18:35:17 +04:00
Yabin Ma
5f21c8bb1d
NET-1778: scale test code changes (#3203)
* comment ACL call and add debug message

* add cache for network nodes

* fix load node to network cache issue

* add peerUpdate call 1 min limit

* add debug log for scale test

* release maps

* avoid default policy for node

* 1 min limit for peerUpdate trigger

* mq options

* Revert "mq options"

This reverts commit 10b93d0118.

* set peerUpdate run in sequence

* update for emqx 5.8.2

* remove batch peer update

* change the sleep to 10 millisec to avoid timeout

* add compress and change encrypt for peerUpdate message

* add mem profiling and automaxprocs

* add failover ctx mutex

* ignore request to failover peer

* remove code without called

* remove debug logs

* update emqx to v5.8.2

* change broker keepalive

* add OLD_ACL_SUPPORT setting

* add host version check for message encrypt

* remove debug message

* remove peerUpdate call control

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
2024-12-10 10:15:31 +04:00
abhishek9686
348a4d963b comment out default pro trial 2024-11-11 09:48:18 +04:00
abhishek9686
32dfcecdff add security to cpu profile 2024-11-08 10:36:25 +04:00
abhishek9686
16b693815f add cpu profiling endpoint 2024-11-07 18:18:08 +04:00
Abhishek K
2e8d95e80e
NET-1227: User Mgmt V2 (#3055)
* user mgmt models

* define user roles

* define models for new user mgmt and groups

* oauth debug log

* initialize user role after db conn

* print oauth token in debug log

* user roles CRUD apis

* user groups CRUD Apis

* additional api checks

* add additional scopes

* add additional scopes url

* add additional scopes url

* rm additional scopes url

* setup middlleware permission checks

* integrate permission check into middleware

* integrate permission check into middleware

* check for headers for subjects

* refactor user role models

* refactor user groups models

* add new user to pending user via RAC login

* untracked

* allow multiple groups for an user

* change json tag

* add debug headers

* refer network controls form roles, add debug headers

* refer network controls form roles, add debug headers

* replace auth checks, add network id to role model

* nodes handler

* migration funcs

* invoke sync users migration func

* add debug logs

* comment middleware

* fix get all nodes api

* add debug logs

* fix middleware error nil check

* add new func to get username from jwt

* fix jwt parsing

* abort on error

* allow multiple network roles

* allow multiple network roles

* add migration func

* return err if jwt parsing fails

* set global check to true when accessing user apis

* set netid for acls api calls

* set netid for acls api calls

* update role and groups routes

* add validation checks

* add invite flow apis and magic links

* add invited user via oauth signup automatically

* create invited user on oauth signup, with groups in the invite

* add group validation for user invite

* update create user handler with new role mgmt

* add validation checks

* create user invites tables

* add error logging for email invite

* fix invite singup url

* debug log

* get query params from url

* get query params from url

* add query escape

* debug log

* debug log

* fix user signup via invite api

* set admin field for backward compatbility

* use new role id for user apis

* deprecate use of old admin fields

* deprecate usage of old user fields

* add user role as service user if empty

* setup email sender

* delete invite after user singup

* add plaform user role

* redirect on invite verification link

* fix invite redirect

* temporary redirect

* fix invite redirect

* point invite link to frontend

* fix query params lookup

* add resend support, configure email interface types

* fix groups and user creation

* validate user groups, add check for metrics api in middleware

* add invite url to invite model

* migrate rac apis to new user mgmt

* handle network nodes

* add platform user to default role

* fix user role migration

* add default on rag creation and cleanup after deletion

* fix rac apis

* change to invite code param

* filter nodes and hosts based on user network access

* extend create user group req to accomodate users

* filter network based on user access

* format oauth error

* move user roles and groups

* fix get user v1 api

* move user mgmt func to pro

* add user auth type to user model

* fix roles init

* remove platform role from group object

* list only platform roles

* add network roles to invite req

* create default groups and roles

* fix middleware for global access

* create default role

* fix nodes filter with global network roles

* block selfupdate of groups and network roles

* delete netID if net roles are empty

* validate user roles nd groups on update

* set extclient permission scope when rag vpn access is set

* allow deletion of roles and groups

* replace _ with - in role naming convention

* fix failover middleware mgmt

* format oauth templates

* fetch route temaplate

* return err if user wrong login type

* check user groups on rac apis

* fix rac apis

* fix resp msg

* add validation checks for admin invite

* return oauth type

* format group err msg

* fix html tag

* clean up default groups

* create default rag role

* add UI name to roles

* remove default net group from user when deleted

* reorder migration funcs

* fix duplicacy of hosts

* check old field for migration

* from pro to ce make all secondary users admins

* from pro to ce make all secondary users admins

* revert: from pro to ce make all secondary users admins

* make sure downgrades work

* fix pending users approval

* fix duplicate hosts

* fix duplicate hosts entries

* fix cache reference issue

* feat: configure FRONTEND_URL during installation

* disable user vpn access when network roles are modified

* rm vpn acces when roles or groups are deleted

* add http to frontend url

* revert crypto version

* downgrade crytpo version

* add platform id check on user invites

---------

Co-authored-by: the_aceix <aceixsmartx@gmail.com>
2024-08-20 17:08:56 +05:30
Sayan Mallick
c551c487ca
New Docs (#3034)
* New Docs

CSS update and Dockerfile to include docs folder

flash of unrendered text fix

markdown docs

ignore docs/docs.go

improving the docs generation

github actions for docs generation

go runner version fix

updated docs.yml

update repo action updated

updated actions and dns docs

dns complete

More docs update

Complete docs and updated workflow

Update documentation Tue Aug  6 11:17:42 UTC 2024

Update documentation Thu Aug  8 12:26:57 UTC 2024

clean up

clean up

Dockerfile clean up

Updated workflow

Updated workflow

Update docs.yml

Update docs.yml

* requested changes

* changed ingress gateway to remote access gateway
2024-08-15 11:55:01 +05:30
Max Ma
65faf73fe9
NET-1226: Scalability Improvements (#2987)
* add api to check if failover node existed

* remove 5 minute peerUpdate

* update peerUpdate to trigger pull

* update Action name to SignalPull

* revert the peerUpdate from SignalPull

* fix getfailover error issue

* rm acls creation for on-prem emqx

* remove use of acls

* add additional broker status field on status api

* NET-1165: Remove creation of acls on emqx (#2996)

* rm acls creation for on-prem emqx

* remove use of acls

* add additional broker status field on status api

* comment out mq reconnect logic

* configure mq conn params

* add metric_interval in ENV for publishing metrics

* add metric_interval in ENV for publishing metrics

* update PUBLISH_METRIC_INTERVAL env name

* revert the mq setttings back

* fix error nil issue

---------

Co-authored-by: abhishek9686 <abhi281342@gmail.com>
Co-authored-by: Abhishek K <32607604+abhishek9686@users.noreply.github.com>
2024-07-09 18:56:55 +05:30
abhishek9686
b2361be29b track failover usage 2024-04-17 17:22:47 +05:30
abhishek9686
4fe282d5d7 add trial license status to server status api 2024-01-23 10:34:45 +05:30
abhishek9686
1f44817e64 add tenantid check 2024-01-23 08:33:17 +05:30
abhishek9686
17980cc222 get trial end date only if license key is missing 2024-01-20 01:39:31 +05:30
abhishek9686
7a39ef3e4e add trial end date to server status api 2024-01-20 01:30:09 +05:30
Gabriel de Souza Seibel
944405f0b7
[ACC-339] Add route to restart server (#2672)
* Add route to restart server with SIGINT

* Respond with 200 in /api/server/health DELETE

* Log before sending SIGINT

* Change route to shutdown
2023-11-22 12:02:11 +04:00
Gabriel de Souza Seibel
001442e75e
[NET-493] - Additional data fields to send to account management (#2583)
* Send relays and internet gws count to amb

* Unify usage funcs in logic pkg

* Fix ee build

* Revert "Unify usage funcs in logic pkg"

This reverts commit 28afc91f7e.

* Add more resource metrics on getUsage handler

* Use pro module's function to get server usage on lic val

* Move GetRelays to pro package
2023-10-06 10:22:58 +04:00
Aceix
fde74e0627
fix(NET-581): add tenant tier to status api (#2558)
* fix(NET-581): add tenant tier to status api

* fix(NET-581): make tenant tier boolean in status api

* fix(NET-581): rename EE to Pro
2023-09-11 09:39:33 +05:30
Gabriel de Souza Seibel
1a1ba1ccf4
[NET-546] Move ee code to ee package, unify ee status and terminology (#2538)
* Move ee code to ee package and unify ee status to IsPro

* Consolidate naming for paid/professional/enterprise version as "pro". Notes:

- Changes image tags
- Changes build tags
- Changes package names
- Doesn't change links to docs that mention "ee"
- Doesn't change parameters sent to PostHog that mention "ee"

* Revert docker image tag being -pro, back to -ee

* Revert go build tag being pro, back to ee

* Add build tags for some ee content

* [2] Revert go build tag being pro, back to ee

* Fix test workflow

* Add a json tag to be backwards compatible with frontend "IsEE" check

* Add a json tag for the serverconfig struct for IsEE

* Ammend json tag to Is_EE

* fix ee tags

---------

Co-authored-by: Abhishek Kondur <abhi281342@gmail.com>
2023-09-01 07:42:05 +05:30
Gabriel de Souza Seibel
8ce7da2ce9
[NET-494 / ACC-322] New free tier limits (#2495)
* Rename var

* Rename consts and use iota

* Use switch instead of repeated else if

* Rename limits related vars

* Introduce new free tier limits

* Measure new limits and report on license validation

* Separate usage and limits, have new ones

* Don't check for hosts and clients limits, but for machines instead

* Error on egress creation @ free tier w/ internet gateways

* Remove clients and hosts limit from code

* Rename var

* Rename consts and use iota

* Use switch instead of repeated else if

* Rename limits related vars

* Introduce new free tier limits

* Measure new limits and report on license validation

* Separate usage and limits, have new ones

* Don't check for hosts and clients limits, but for machines instead

* Error on egress creation @ free tier w/ internet gateways

* Remove clients and hosts limit from code
2023-08-08 23:17:49 +05:30
Gabriel de Souza Seibel
922e7dbf2c
[NET-404] Run in limited mode when ee checks fail (#2474)
* Add limited http handlers functionality to rest handler

* Export ee.errValidation (ee.ErrValidation)

* Export a fatal error handled by the hook manager

* Export a new status variable for unlicensed server

* Mark server as unlicensed when ee checks fail

* Handle license validation failures with a (re)boot in a limited state

* Revert "Export a fatal error handled by the hook manager"

This reverts commit 069c21974a8d36e889c73ad78023448d787d62a5.

* Revert "Export ee.errValidation (ee.ErrValidation)"

This reverts commit 59dbab8c79773ca5d879f28cbaf53f3dd4297b9b.

* Revert "Add limited http handlers functionality to rest handler"

This reverts commit e2f1f28facaca54713db76a588839cd2733cf673.

* Revert "Handle license validation failures with a (re)boot in a limited state"

This reverts commit 58cfbbaf522a1345aac1fa67964ebff0a6d60cd8.

* Revert "Mark server as unlicensed when ee checks fail"

This reverts commit 77c6dbdd3c9cfa6e7d6becedef6251e8617ae367.

* Handle license validation failures with a middleware

* Forbid responses if unlicensed ee and not in status api

* Remove unused func
2023-08-03 12:16:58 +05:30
Abhishek K
230e062c84
GRA-1298: License check changes, free tier limits for saas (#2418)
* set free tier limits through config

* add host limit to config

* check for host limit on free tier

* fix license validation, replace node limit with hosts

* add hosts to telemetry data

* debug init

* validate license every 1hr

* hook manager, api to fetch server usage

* hook manager, server usage api

* encode json server usage api

* update ngork url

* update license validation endpoint

* avoid setting limits on eer

* adding hotfix

* correct users limits env var

* add comments to exported funcs

---------

Co-authored-by: afeiszli <alex.feiszli@gmail.com>
2023-06-28 20:33:06 +05:30
Abhishek K
6c2582651c
Net-137: Node relays (#2401)
* revert relays

* initial relay commit

* get relayed allowed ips

* add more relay validation checks, peer logic cleanup

* rm relayed nodes from relay node when relayed node is deleted

* fix egress updates for relayed nodes

* rm unused func

* remove  debug logs

* avoid adding egress ranges on the relayed gw node

---------

Co-authored-by: Matthew R Kasun <mkasun@nusak.ca>
2023-06-19 12:26:14 -04:00
Tobias Cudnik
723375b334
NET-152 enrollment keys for non admins (#2346)
* return 401 instead of 403

* fixed http.StatusForbidden

* Tagged build version (temp)

* Unauthorized_Err when applicable

* untagged version

* fixed PUT /api/users/networks/user1

* - expired token redirs to login
- added `/api/enrollment_keys` for non-admins
- unit test for enrollment keys for non-admins

* handle user perms in `/hosts`

* removed debug

* misc

* - support masteradmin
- return hosts with partial access

* added `ismaster` to middleware
2023-05-31 13:11:54 +05:30
Tobias Cudnik
3a4363c890
NET-163: Return 403 instead of 401 (#2326)
* return 401 instead of 403

* fixed http.StatusForbidden

* Tagged build version (temp)

* Unauthorized_Err when applicable

* untagged version
2023-05-25 09:40:39 -04:00
0xdcarns
7a2c225eb1 added mq and database connected funcs and endpoint 2023-02-06 12:09:21 -05:00
walkerwmanuel
6611941ca2 Replaced string methods with HTTP consts 2022-12-23 09:49:08 -05:00
Abhishek Kondur
a04549eebe rm register endpoint 2022-09-30 23:19:33 +05:30
Abhishek Kondur
faf7addcf5 Merge branch 'bugfix_v0.16.1_ee_checks' of https://github.com/gravitl/netmaker into feature_dyn_sec_latest 2022-09-26 18:09:34 +05:30
Abhishek Kondur
f632bec11d merge conflicts resolved 2022-09-26 17:47:20 +05:30
Abhishek Kondur
cbcb046139 added server health check api 2022-09-26 16:55:33 +05:30
Matthew R. Kasun
5f7d6ea1ef logic.Is_EE --> systemcfg.Is_EE 2022-09-25 07:11:26 -04:00
afeiszli
d19cac35fa removed remove network 2022-09-19 12:59:37 -04:00
Alex Feiszli
53d2250049
Merge branch 'develop' into bugfix_v0.16.0_swagger_sections 2022-09-19 12:55:32 -04:00
cameronts
4cc32a5b50 Fix sections for /server/ and /users/ paths in Swagger docs 2022-09-15 05:47:48 -07:00
afeiszli
b670755cce refactoring for ee 2022-09-14 13:26:31 -04:00
afeiszli
4917c457db added ee check 2022-09-13 15:44:45 -04:00
cameronts
661290c84d Update swagger docs 2022-09-10 21:51:59 -07:00
cameronts
cb23b871dd Added swagger for API docs. 2022-09-06 05:20:24 -07:00
Abhishek Kondur
0c9f3cf251 GRA-414: added additional logs for sever controllers 2022-07-12 20:03:14 +04:00
afeiszli
f28d361bea refactoring cert logic to use database 2022-06-30 22:30:28 -04:00
afeiszli
b281047168 adding functions to handle certs in DB 2022-06-30 21:56:11 -04:00
afeiszli
8d422526b3 refactoring servercfg 2022-05-31 12:07:56 -04:00
afeiszli
e4189ad8bd refactoring serverinfo logic 2022-05-31 08:42:12 -04:00
afeiszli
0865a535c7 changing broker retrieval/setting logic 2022-05-30 12:39:33 -04:00
Matthew R. Kasun
f5aa1d1e62 reorder register/join 2022-04-25 10:38:29 -04:00
Matthew R. Kasun
955e0e544c join as api call
Signed-off-by: Matthew R. Kasun <mkasun@nusak.ca>
2022-04-20 15:46:19 -04:00
Matthew R. Kasun
2b1f20e94b changes from code review
Signed-off-by: Matthew R. Kasun <mkasun@nusak.ca>
2022-04-18 18:21:55 -04:00
Matthew R. Kasun
1cbbd52e7d fixed comment
Signed-off-by: Matthew R. Kasun <mkasun@nusak.ca>
2022-04-18 09:47:01 -04:00