SIP/WebRTC Messaging Server
Sylk Server provides online and offline messaging services for standard SIP end-points and WebRTC clients that support SylkRTC API.
The functionality is similar to what WhatsApp and Telegram offer with the difference that both client and server software are fully open source and based on open standards. Reference Web, Mobile, Desktop implementations and a live service are available for testing and interoperability purposes.
Sylk Server maintains a journal for each user with the messages and their modifications encapsulated using the following standards:
SIP Message method - RFC3261
CPIM Message Format - RFC3862
Instant Message Disposition Notification (IMDN) - RFC5438
SIP offline messages can by synced between all devices by applying the journal of operations for each user.
For real time synchronization of online devices, standard SIP parallel forking is used.
The storage is transparent for end-users and it is up to the end-user devices to encrypt the payloads before sending them out by using an OpenPGP implementation.
Sylk Server has support for managing the public and private keys for PGP encrypted messages.
Features
Message replication between multiple devices
Offline storage when devices are not online
Message delivery and display reports
Message deletion, partial or per contact
Tracks remaining unread messages
Incremental journal per account
Distributed storage support using Cassandra
SIP and WebRTC APIs
Mobile push notifications
PGP public/private keys management
API
The functions are documented in this JavaScript API:
Sample clients
Complete fully featured clients implementing Sylk messaging API:
Blink for Mac: https://github.com/AGProjects/blink-cocoa
Sylk Mobile: https://github.com/AGProjects/sylk-mobile
Sylk Web and Desktop: https://github.com/AGProjects/sylk-webrtc
sip-session3 terminal client: https://sipsimpleclient.org/testing/
Content types
Sylk Server handle in special ways different content types.
application/sylk-api-token
Allows a SIP device to retrieve an authentication token that can be later used for authorization to retrieve the journal using HTTP requests.
Example:
SENDING: Packet 60, +0:01:52.503136 2021-11-15 14:33:07.194842: 192.168.1.11:50768 -(SIP over TLS)-> 85.17.186.23:5061 MESSAGE sip:ag@sip2sip.info SIP/2.0 Via: SIP/2.0/TLS 192.168.1.11:50768;rport;branch=z9hG4bKPj8MUCkGXTF7nXCTYxcMHYL5tzC8p2j-;alias From: <sip:ag@sip2sip.info>;tag=ePBmQXDzJCFfRZR3FekS3mTjCsapf To: <sip:ag@sip2sip.info> Call-ID: X6saJssTX.wmSZYDf2GrZXOAoGMiEBh0 User-Agent: Blink 8.7.0 (MacOSX) Content-Type: application/sylk-api-token Content-Length: 14 I need a token
Response:
RECEIVED: Packet 62, +0:01:52.816791
2021-11-15 14:33:07.508497: 85.17.186.23:5060 -(SIP over TCP)-> 192.168.1.11:50735
MESSAGE sip:46139205@192.168.1.11:50685;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 85.17.186.23:5060;branch=z9hG4bKf3d6.33c61951.0;i=57fa5934
Via: SIP/2.0/TCP 85.17.186.26:35305;received=85.17.186.26;rport=35305;branch=z9hG4bKPj5eb777ff-0026-44b0-b17b-9d468fe76931;alias
From: "SylkServer" <sip:sylkserver@85.17.186.26>;tag=7449abda-1da-4df3-b8d1-0c94cb031f18
To: <sip:ag@sip2sip.info>
Call-ID: 3507f82d-f027-4d3a-bfb6-3efb4655caa1
User-Agent: SylkServer-6.0.0
Content-Type: application/sylk-api-token
Content-Length: 153
{"token": "iKQI7svOtyGkfsfsfsh9qGstVUOnTgrFxpo1ik40", "url": "https://webrtc-gateway.sipthor.net:9943/webrtcgateway/messages/history/ag@sip2sip.info"}
By sending a GET to the URL, one can retrieve the journal since the last id.
if last_id:
url = "%s/%s" % (url, account.sms.history_last_id)
req = urllib.request.Request(url, method="GET")
req.add_header('Authorization', 'Apikey %s' % account.sms.history_token)
Typically the request is sent every time the device connects to the network and then saves the last id that can be used later to resume the journal processing.
application/sylk-api-pgp-key-lookup
Allows a SIP device to retrieve an public pgp key stored on the server. The To-Header is used to specify the user for which the key needs to be found
Example:
MESSAGE sip:ag@sip2sip.info SIP/2.0 Via: SIP/2.0/TLS 192.168.1.11:50768;rport;branch=z9hG4bKPj8MUCkGXTF7nXCTYxcMHYL5tzC8p2j-;alias From: <sip:ag@sip2sip.info>;tag=ePBmQXDzJCFfRZR3FekS3mTjCsapf To: <sip:tijmen@sip2sip.info> Call-ID: X6saJssTX.wmSZYDf2GrZXOAoGMiEBh0 User-Agent: Blink 8.7.0 (MacOSX) Content-Type: application/sylk-api-pgp-key-lookup Content-Length: 14 I need a public key
Response:
MESSAGE sip:46139205@192.168.1.11:50685;transport=tcp SIP/2.0 Via: SIP/2.0/TCP 85.17.186.23:5060;branch=z9hG4bKf3d6.33c61951.0;i=57fa5934 Via: SIP/2.0/TCP 85.17.186.26:35305;received=85.17.186.26;rport=35305;branch=z9hG4bKPj5eb777ff-0026-44b0-b17b-9d468fe76931;alias From: <sip:tijmen@sip2sip.info>;tag=7449abda-1da-4df3-b8d1-0c94cb031f18 To: <sip:ag@sip2sip.info> Call-ID: 3507f82d-f027-4d3a-bfb6-3efb4655caa1 User-Agent: SylkServer-6.0.0 Content-Type: text/pgp-public-key Content-Length: .... -----BEGIN PGP PUBLIC KEY BLOCK----- xsFNBGDQz0cBEACw7ZCd2CIw5udWY5VOV4ZrzvyIdt8idBoUqbUJ6Lm55KVMj7Kh ..... cduwml6F7g== =Xp4B -----END PGP PUBLIC KEY BLOCK-----
text/pgp-public-key
Once a user sends out a message with such payload, it will be stored in the user profile and end-points that implement public key fetch function can retrieve the key before starting a conversation.
text/pgp-private-key
This is used for end-points to replicate their private keys, so multiple devices can by synchronized.
To replicate messages on multiple devices you need the same private key on all of them.
To be compatible with existing clients the message must be addressed to the same user and the payload must consist of the user PGP public key in clear text followed by the PGP encrypted with a symmetric code private key.
Example:
SENDING: Packet 88, +0:35:02.954481 2021-11-15 15:06:17.646187: 192.168.1.11:51541 -(SIP over TLS)-> 85.17.186.23:5061 MESSAGE sip:ubuntu@test.com SIP/2.0 Via: SIP/2.0/TLS 192.168.1.11:51541;rport;branch=z9Xb--OH3b91;alias Max-Forwards: 70 From: <sip:ubuntu@test.com>;tag=pPOSjBJx2nm9ZG8DZX-ObDtCfDaqJBDk To: <sip:ubuntu@test.com> Call-ID: bhNsVTpEoFNNqXtNcpm CSeq: 52490 MESSAGE User-Agent: Blink 8.7.0 (MacOSX) Content-Type: text/pgp-private-key Content-Length: 5378 -----BEGIN PGP PUBLIC KEY BLOCK----- xsFNBGDQz0cBEACw7ZCd2CIw5udWY5VOV4ZrzvyIdt8idBoUqbUJ6Lm55KVMj7Kh 6Lu13BKfV3m04Q3L3b5KboqIsbJLivlpg2YX+vOAAzEDE9BTZN0o+orDWxP1Dz9U LvtloSRoNK1NZj+KkbKfRXUZCMnyXDiN64cYSFLGp0esT/DhyfI09p63dqegl+Ut uenVLzne9PkpdewS0OUmfKUGBY86SPUewwBkFFVxYyIWQu+yJLfMt7jxTHnD3dcT mDaRljMZj6qhzQ2F2sc68KHy2FIeXR9kOxMq8T6aEw0JwAA4rLWWCT6MLEtjDPng cduwml6F7g== =Xp4B -----END PGP PUBLIC KEY BLOCK----- -----BEGIN PGP MESSAGE----- O+rMXHZq7rhMlYyBYJFSgVZTn2t58s/3NkTVewwvgstl10BUDgn9zBggElo86duB 2fqwEGCYSny1CLfWtMe/8GBrLd/pCpAWVlzsaI/s/7QbLqW8d0nq1uft8ubRvyWb wIl0lLwuSReeyNcFHw53jpumFQbXYtZRM8UhhKmJoeKwZh++AIYUdG5OlPUb32hF 8mhmr1m1rhxPJTWQvaCsYg== =sZwQ -----END PGP MESSAGE-----
The receiving end-point must decrypt the message and save the PGP keys for future use.
It is advisable to use more than one device and replicate the private key to all of them so that there is still a copy of the private key available in case the main device is lost.
Journal
The journal consists of a json file with entries for distinct operations:
Incoming messages
Outgoing messages
Key exchange operations
IMDN notifications
Message removal
Conversation read
Conversation removal
Check the source code of the sample clients for working examples of parsing the journal.
Setup
Sylk Server
Install Sylk Server and enable webrtc_gateway application. Initialize the storage.
There is nothing else to configure (zero configuration).
Database management
Using provided sylk-db maintenance tool one can see details about the storage usage or show information about individual accounts.
agp@sylkserver-test:~$sylk-db show test@sylk.link ...... Reading configuration from /etc/sylkserver/config.ini ...... ...... ************************ SylkServer - Show account data ************************ ...... ...... Reading storage configuration from /etc/sylkserver/webrtcgateway.ini ...... New Cassandra host <Host: 10.0.0.146:9042 GlobalSwitch> discovered ...... Server has keyspace sipthor with replication strategy: NetworkTopologyStrategy ...... ...... -------------------------------------------------------------------------------- ...... test@sylk.link ...... -------------------------------------------------------------------------------- ...... Message storage is enabled ...... Last login at: 2021-11-12 12:08:45.356000 ...... ...... 459 messages stored ...... ...... Text Messages: 134 ...... IMDN messages: 308 ...... Other Messages: 17 ...... ...... Unread text Messages: 1 ...... ...... 1 push token(s) stored ...... ...... App: com.agprojects.sylk-ios.prod ...... Device ID: 2EBAAC58-D245-48EE-82E6-53FD4EF0D57 ...... Token: cac10301b55661d46b3fbb ... ...... ...... 1 public key(s) stored ...... ...... -----BEGIN PGP PUBLIC KEY BLOCK----- ...... Version: fast-openpgp ...... ...... xsFABGExS1wBEAC660kacrxXINz/1DtZ94p0mJuChgnkKShg0n6cQDbSN7vh2/hs ...... TBYf94eQkYT2gU0ZRBcsJinC6opnQHLTEk3grBHE2fJGU6czoEpej1HZHegVa/D ...... orfZieBFmmH4Sqk8j6I6mNg+UQ2RzI8WAUcXv0hOc9OQzZw3NhoDkDJGXF9bsNTk ...... QoHjlVxX7XFJQui8AecMDrD/h11iF01XuwTOchDCcU5waTlXztyNn2FJVufgX1gd ...... B+kWgOMFu0VWC++N3132KwA4uJb3eoaocZEmRKEM ...... =1sgu ...... -----END PGP PUBLIC KEY BLOCK-----
OpenSIPS
Sylk Server relies on a SIP Proxy for dispatching its messages.
Bellow is the logic code for OpenSIP that implements forking of messages to Sylk Server:
if ($rm=="MESSAGE") {
if (db_does_uri_exist($ru, "subscriber")) {
t_on_failure("FAILED_MESSAGE");
}
$avp(sip_application_type) = "message";
$avp(can_uri) = $tu;
$avp(source_ip) = $si;
$avp(source_port) = $sp;
$avp(sip_proxy_ip) = "192.168.1.100";
if (not has_totag()) {
t_on_reply("DIALOG_REPLY");
if (loose_route()) {
xlog("L_WARN", "[CONFIG] WARNING: Incorrectly formatted $rm request. Rejected with 400. ($ci)\n");
sl_send_reply(400, "Incorrectly formatted request");
return;
}
if (is_from_local()) {
if (isflagset("DO_AUTHENTICATE_FLAG")) {
if (not proxy_authorize("", "subscriber")) {
xlog("L_INFO", "[CONFIG] Asking for $fU@$fd credentials ($ci)\n");
proxy_challenge("", "auth,auth-int");
return;
} else if ($au != $fU) {
xlog("L_INFO", "[CONFIG] Rejected with 403 because $au != $fU ($ci)\n");
sl_send_reply(403, "Username!=From not allowed ($au!=$fU)");
return;
}
# Hide auth credentials to downstream routers
consume_credentials();
}
}
if ($avp(source_ip) != "10.0.0.1") {
# Inform Sylk to replicate an outgoing message
append_hf("X-Sylk-From-Sip: yes\r\n");
}
if (is_uri_host_local()) {
if (db_does_uri_exist($ru, "subscriber")) {
if ($hdr(Content-Type) == "application/sylk-api-token") {
# api-token requests from ourselves must be only sent to Sylk Server
if ($avp(source_ip) != "10.0.0.1") {
xlog("L_INFO", "[CONFIG] Route $rm $hdr(Content-Type) $ru to Sylk Server 10.0.0.1\n");
$du = "10.0.0.1";
if (not t_relay()) {
sl_reply_error();
}
exit;
}
# replies from Sylk Server with token will be routed only to the end-points
} else if (is_present_hf("X-Replicated-Message")) {
xlog("L_DBG", "[CONFIG] Skip forking $hdr(Content-Type) X-Replicated-Message MESSAGE to Sylk server ($ci)\n");
exit;
}
# Should we fork message to ourselves to have a copy on other online devices
$var(must_fork) = 1;
if ($var(must_fork) && $hdr(Content-Type) == "application/im-iscomposing+xml") {
$var(must_fork) = 0;
}
if ($var(must_fork) && search_body("message/imdn+xml")) {
$var(must_fork) = 0;
}
if ($var(must_fork) && $ru != 'sip:$fu@$fd') {
$var(must_fork) = 0;
}
if ($var(must_fork) && is_present_hf("X-Replicated-Message")) {
$var(must_fork) = 0;
}
$var(orig_ru) = $ru;
if (lookup("location")) {
append_branch();
xlog("L_INFO", "[CONFIG] Fork $rm $hdr(Content-Type) $ru to Sylk Server 10.0.0.1\n");
seturi("10.0.0.1");
append_hf("X-Sylk-App: webrtcgateway\r\n");
if ($var(must_fork)) {
$var(fu) = "sip:" + $fU + "@" + $fd;
xlog("L_INFO", "[CONFIG] Forking MESSAGE for $var(orig_ru) to myself $var(fu) ($ci)\n");
append_branch();
seturi($var(fu));
append_hf("X-Replicated-Message: yes\r\n");
}
if (not t_relay()) {
sl_reply_error();
}
exit;
} else {
xlog("L_INFO", "[CONFIG] $ru is not yet online ($ci)\n");
xlog("L_INFO", "[CONFIG] Forward $rm $hdr(Content-Type) for $var(orig_ru) to Sylk Server 10.0.0.1\n");
$ru = "10.0.0.1";
append_hf("X-Sylk-App: webrtcgateway\r\n");
if ($var(must_fork)) {
$var(fu) = "sip:" + $fU + "@" + $fd;
xlog("L_INFO", "[CONFIG] Forking MESSAGE for $var(orig_ru) to myself $var(fu) ($ci)\n");
append_branch();
seturi($var(fu));
append_hf("X-Replicated-Message: yes\r\n");
}
sl_send_reply(480, "User not online");
return;
}
} else {
sl_send_reply(404, "User not found");
return;
}
}
} else {
# In-dialog MESSAGE
if (not loose_route()) {
# Only relay in-dialog requests that were previously Record-Routed by us
sl_send_reply(400, "In-dialog MESSAGE rejected");
exit;
}
}
- Full instructions for installing this configuration are available at: