Bethuel Mmbaga 4cdb2e533a [management] Refactor users to use store methods (#2917)
* Refactor setup key handling to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add lock to get account groups

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add check for regular user

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* get only required groups for auto-group validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add account lock and return auto groups map on validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor account peers update

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor groups to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* refactor GetGroupByID and add NewGroupNotFoundError

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add AddPeer and RemovePeer methods to Group struct

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Preserve store engine in SqlStore transactions

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Run groups ops in transaction

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix missing group removed from setup key activity

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor posture checks to remove get and save account

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix refactor

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix sonar

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Change setup key log level to debug for missing group

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Retrieve modified peers once for group events

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor policy get and save account to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Retrieve policy groups and posture checks once for validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix typo

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add policy tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor anyGroupHasPeers to retrieve all groups once

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor dns settings to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add account locking and merge group deletion methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor name server groups to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add peer store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor ephemeral peers

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add lock for peer store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor peer handlers

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor peer to use store methods

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix typo

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add locks and remove log

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* run peer ops in transaction

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* remove duplicate store method

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix peer fields updated after save

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* add tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Use update strength and simplify check

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* prevent changing ruleID when not empty

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* prevent duplicate rules during updates

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix lint

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor auth middleware

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor account methods and mock

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor user and PAT handling

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Remove db query context and fix get user by id

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix database transaction locking issue

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Use UTC time in test

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add account locks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix prevent users from creating PATs for other users

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add store locks and prevent fetching setup keys peers when retrieving user peers with empty userID

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add missing tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor test names and remove duplicate TestPostgresql_SavePeerStatus

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add account locks and remove redundant ephemeral check

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Retrieve all groups for peers and restrict groups for regular users

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix store tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* use account object to get validated peers

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Improve peer performance

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Get account direct from store without buffer

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Add get peer groups tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Adjust benchmarks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Adjust benchmarks

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* [management] Update benchmark workflow (#3181)

* update local benchmark expectations

* update cloud expectations

* Add status error for generic result error

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Use integrated validator direct

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* update expectations

* update expectations

* update expectations

* Refactor peer scheduler to retry every 3 seconds on errors

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* update expectations

* fix validator

* fix validator

* fix validator

* update timeouts

* Refactor ToGroupsInfo to process slices of groups

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* update expectations

* update expectations

* update expectations

* Bump integrations version

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor GetValidatedPeers

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* go mod tidy

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Use peers and groups map for peers validation

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* remove mysql from api benchmark tests

* Fix merge

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix blocked db calls on user auto groups update

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* update expectations

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* update expectations

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Skip user check for system initiated peer deletion

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Remove context in db calls

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* update expectations

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* [management] Improve group peer/resource counting (#3192)

* Fix sonar

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Adjust bench expectations

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Rename GetAccountInfoFromPAT to GetTokenInfo

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Remove global account lock for ListUsers

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* build userinfo after updating users in db

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* [management] Optimize user bulk deletion  (#3315)

* refactor building user infos

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* fix tests

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* remove unused code

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Refactor GetUsersFromAccount to return a map of UserInfo instead of a slice

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Export BuildUserInfosForAccount to account manager

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Fetch account user info once for bulk users save

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Update user deletion expectations

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Set max open conns for activity store

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

* Update bench expectations

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>

---------

Signed-off-by: bcmmbaga <bethuelmbaga12@gmail.com>
Co-authored-by: Pascal Fischer <32096965+pascal-fischer@users.noreply.github.com>
Co-authored-by: Pascal Fischer <pascal@netbird.io>
Co-authored-by: Pedro Costa <550684+pnmcosta@users.noreply.github.com>
2025-02-17 21:43:12 +03:00
2022-10-22 16:19:16 +02:00
2024-08-14 13:30:10 +03:00
2024-06-13 13:24:24 +02:00
2024-10-30 17:18:27 +01:00
2022-12-02 13:54:22 +01:00

Start using NetBird at netbird.io
See Documentation
Join our Slack channel


NetBird combines a configuration-free peer-to-peer private network and a centralized access control system in a single platform, making it easy to create secure private networks for your organization or home.

Connect. NetBird creates a WireGuard-based overlay network that automatically connects your machines over an encrypted tunnel, leaving behind the hassle of opening ports, complex firewall rules, VPN gateways, and so forth.

Secure. NetBird enables secure remote access by applying granular access policies while allowing you to manage them intuitively from a single place. Works universally on any infrastructure.

Open-Source Network Security in a Single Platform

netbird_2

NetBird on Lawrence Systems (Video)

Watch the video

Key features

Connectivity Management Security Automation Platforms
  • - [x] Kernel WireGuard
  • - [x] Linux
  • - [x] Peer-to-peer connections
  • - [x] Auto peer discovery and configuration
  • - [x] Mac
  • - [x] Connection relay fallback
  • - [x] Windows
  • - [x] IdP groups sync with JWT
  • - [x] Android
  • - [x] NAT traversal with BPF
  • - [x] Peer-to-peer encryption
  • - [x] iOS
  • - [x] OpenWRT
  • - [x] Periodic re-authentication
    • - [x] Docker

    Quickstart with NetBird Cloud

    Quickstart with self-hosted NetBird

    This is the quickest way to try self-hosted NetBird. It should take around 5 minutes to get started if you already have a public domain and a VM. Follow the Advanced guide with a custom identity provider for installations with different IDPs.

    Infrastructure requirements:

    • A Linux VM with at least 1CPU and 2GB of memory.
    • The VM should be publicly accessible on TCP ports 80 and 443 and UDP ports: 3478, 49152-65535.
    • Public domain name pointing to the VM.

    Software requirements:

    • Docker installed on the VM with the docker-compose plugin (Docker installation guide) or docker with docker-compose in version 2 or higher.
    • jq installed. In most distributions Usually available in the official repositories and can be installed with sudo apt install jq or sudo yum install jq
    • curl installed. Usually available in the official repositories and can be installed with sudo apt install curl or sudo yum install curl

    Steps

    • Download and run the installation script:
    export NETBIRD_DOMAIN=netbird.example.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash
    
    • Once finished, you can manage the resources via docker-compose

    A bit on NetBird internals

    • Every machine in the network runs NetBird Agent (or Client) that manages WireGuard.
    • Every agent connects to Management Service that holds network state, manages peer IPs, and distributes network updates to agents (peers).
    • NetBird agent uses WebRTC ICE implemented in pion/ice library to discover connection candidates when establishing a peer-to-peer connection between machines.
    • Connection candidates are discovered with the help of STUN servers.
    • Agents negotiate a connection through Signal Service passing p2p encrypted messages with candidates.
    • Sometimes the NAT traversal is unsuccessful due to strict NATs (e.g. mobile carrier-grade NAT) and a p2p connection isn't possible. When this occurs the system falls back to a relay server called TURN, and a secure WireGuard tunnel is established via the TURN server.

    Coturn is the one that has been successfully used for STUN and TURN in NetBird setups.

    See a complete architecture overview for details.

    Community projects

    Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases.

    Support acknowledgement

    In November 2022, NetBird joined the StartUpSecure program sponsored by The Federal Ministry of Education and Research of The Federal Republic of Germany. Together with CISPA Helmholtz Center for Information Security NetBird brings the security best practices and simplicity to private networking.

    CISPA_Logo_BLACK_EN_RZ_RGB (1)

    Testimonials

    We use open-source technologies like WireGuard®, Pion ICE (WebRTC), Coturn, and Rosenpass. We very much appreciate the work these guys are doing and we'd greatly appreciate if you could support them in any way (e.g., by giving a star or a contribution).

    WireGuard and the WireGuard logo are registered trademarks of Jason A. Donenfeld.

    Description
    No description provided
    Readme BSD-3-Clause 358 MiB
    Languages
    Go 97.1%
    Shell 1.7%
    HTML 0.7%
    TypeScript 0.4%