See more at the announce forum.
One of the most surprising feedback I got on my "D Cookbook" back in 2015 was people appreciated my simple socket example with Phobos. This was surprising because I didn't spend a lot of time on it and didn't think many people would care!
So today, I'm going to revisit the topic and write a little more detail.
Sockets are used for communication between two computers, or sometimes, two programs on the same computer. They can be used bidirectionally - to both send and receive data - as long as both sides agree on an address and protocol.
Socket types can be generally broken up into categories based on address type and data type. Common address types include IP, IPv6, and Unix sockets. Common data types include streaming data and datagrams. I'll explain these in more detail later.
With Phobos' std.socket, you pass these types to the Socket constructor. AddressFamily defines the address type and SocketType is the data type. You'll see there are other options in there than the ones I'll talk about here, but they are relatively rare for ordinary applications and I don't want to be writing all night.
After creating a socket, the next step depends on your data type: if a datagram type, you will bind to an address and start communicating. If a streaming type, there's a difference between client and server. The server must bind to an address, start listening on that address, then accept connections before communicating. The server connects to an existing server address then can start communicating.
Let's get into the details.
The address type varies with what kind of network you and the computer you want to communicate are on.
Computers communicating across LANs or the Internet will use [AddressFamily.INET ] or AddressFamily.INET6 depending on if the network you are communicating on is IPv6 enabled. I'm going to use plain INET for my example, but they work basically the same way. You can also use these addresses for two applications on the same computer, too. These addresses are domain names or numeric sets combined with a port number. For example dpldocs.info on port 80 or 127.0.0.1 on port 7654.
Two applications on the same computer may perhaps communicate over AddressFamily.UNIX addresses, which are like filenames, for example /tmp/my_socket. (Despite its name, a UNIX address may be used on newer versions of Windows too, but that's fairly new. On Windows, I'd just use an IP address unless you are sure your program needs UNIX addresses and your target computer support it.)
UNIX address sockets are more efficient than IP sockets and have other benefits like being able to communicate local user account information and not be limited by port numbers, but cannot actually communicate over a network.
With Phobos, you construct the addresses differently:
auto unix = new UnixAddress("/tmp/my_socket"); auto ip = new InternetAddress("dpldocs.info", 80);
But once they are constructed, you use them the same way.
Datagrams are packets of data sent and received as a unit. They do not need a connection, meaning you can send and receive as soon as you bind, but there is the possibility that datagrams will get lost on the network and never make it to their destination. They also have a maximum size that the whole network must agree upon (typically <= 4 kilobytes). You might never know that data got lost in transit. Packets may also be rearranged in transit, so ones you send later may arrive first. On the other hand, they tend to be lower latency due to their lightweight nature.
Thus, datagrams' apparent simplicity gets messy if you need more data size or reliability. That's what stream connections are for. But if you can afford to use data, or are using a UNIX address, where there is no actual network to lose your data (though you still must abide by the maximum packet size), the datagram can be useful.
Let's write a pair of example programs to talk to each other. I'll use an IP address here, though both programs will run on your own computer due to the localhost address. If you want to try a UNIX address, simply replace the address constructor line.
First, this program will be set up to receive. It should be running first. This one will wait for a message, then reply to the sender. Make sure you read the inline comments.
1 import std.socket, std.stdio; 2 3 void main() { 4 // to use a UNIX address, change the family here and 5 // then also change the `new InternetAddress` lines to 6 // `new UnixAddress`. 7 auto socket = new Socket(AddressFamily.INET, SocketType.DGRAM); 8 9 // note that this port is explicitly given so we can 10 // refer to it from the other program. 11 socket.bind(new InternetAddress("localhost", 8765)); 12 13 // A common mistake I see among new socket users is not 14 // defining a buffer with a size. ALL buffers with std.socket 15 // must have a size bigger than zero or else it will not work. 16 char[1024] buffer; 17 18 // this will be populated by the receiveFrom function, telling 19 // us from whom the message came. 20 Address from; 21 22 try_again: 23 writeln("Waiting for data..."); 24 25 auto value = socket.receiveFrom(buffer[], from); 26 // the return value might indicate an error, which 27 // is not necessarily fatal - it might just be because 28 // the network was busy and you need to try again later. 29 // I'll write more about this later. 30 if(value == Socket.ERROR) { 31 // Phobos does a bad job exposing errors. It 32 // offers wouldHaveBlocked as a separate flag, 33 // and then just a string for lastSocketError. 34 // 35 // On Posix systems, it can be important to check 36 // for EINTR - that the call was interrupted 37 // and should be retried. 38 // 39 // Many times the operating system can do this 40 // for you, depending on how the signal handler 41 // is set up. But if you aren't automatically 42 // retrying, you need to check that code through 43 // the C errno function. 44 version(Posix) { 45 import core.stdc.errno; 46 if(errno == EINTR) 47 goto try_again; 48 } 49 // otherwise, just throw the string Phobos offers. 50 throw new Exception(lastSocketError()); 51 } 52 53 // if it isn't an error though, it indicates the length of 54 // data received. With datagram packets, they should never 55 // have a value of zero, since that indicates the connection 56 // is closed, and we have no connection anyway! 57 // 58 // So I'll just slice the buffer to get the data which may 59 // be empty. 60 auto data = buffer[0 .. value]; 61 62 writeln("Received: ", data); 63 64 // and let's send a reply: 65 66 auto reply = "got it!"; 67 value = socket.sendTo(reply[], from); 68 // again, check for error 69 if(value == Socket.ERROR) 70 throw new Exception(lastSocketError()); 71 // but otherwise the datagram should be sent in its 72 // entirety in one go. Anything less means the network 73 // has some other size limit and you should probably fix 74 // your code to work with the smaller packets. 75 assert(value == reply.length); 76 }
And now this one will send a packet to the first one, then wait for the reply. Again, be sure to read the comments. To try for yourself, run the first program, then while it is running, run the second program and see them talk to each other.
1 import std.socket, std.stdio; 2 3 void main() { 4 auto socket = new Socket(AddressFamily.INET, SocketType.DGRAM); 5 // note the address here is set to ANY because it will 6 // only be receiving replies to data it sends, and thus 7 // the address does not need to be fixed to something. 8 socket.bind(new InternetAddress(InternetAddress.ADDR_ANY, InternetAddress.PORT_ANY)); 9 10 string dataPacket = "Hello!"; 11 writeln("Sending data..."); 12 // datagrams don't need connections, so we just send 13 // the packet to the address we set up in the other program 14 auto err = socket.sendTo(dataPacket, new InternetAddress("localhost", 8765)); 15 if(err == Socket.ERROR) 16 throw new Exception(lastSocketError()); 17 // same length assumption as described above... 18 assert(err == dataPacket.length); 19 20 // and now we will listen for a reply. All sockets can 21 // be used to both send and receive. 22 char[1024] buffer; 23 Address from; 24 err = socket.receiveFrom(buffer[], from); 25 if(err == Socket.ERROR) 26 throw new Exception(lastSocketError()); 27 28 auto data = buffer[0 .. err]; 29 writeln("Got response: ", data); 30 }
To recap, the steps are:
To solve the reliability problem and size limits incurred by datagrams, stream protocols were developed. On the internet, this is implemented by TCP, the transmission control protocol, so-named because it controls the transmission of data so it arrives in sequence, detects and resends lost data, and can handle arbitrary sizes of data.
This takes a lot of complexity off your hands in many cases, but you also must be able to process data streams instead of just standalone packets, and write a little more code to establish those reliable connections.
The socket API and operating system take care of a lot of details for you, but there are a few more function calls you need and it is extra important to process the return values on send and receive in more detail than on datagrams.
Let's get into the examples. Connection-based streams have very different flows for servers and clients, though once the connection is made, you can communicate each direction equally well.
Assume your data will not be received in a single call and may have boundaries in the middle. Read the inline comments carefully.
1 import std.socket, std.stdio; 2 3 void main() { 4 // like always, first create the socket with the params 5 auto listener = new Socket(AddressFamily.INET, SocketType.STREAM); 6 7 // this is not always the right thing to do, but I often 8 // use it - this tweaks a setting allowing you to reuse 9 // an address that was recently used. Normally, TCP keep 10 // the connection on hold after the program exits for a 11 // minute or two in order to clean up any data that got 12 // delayed on the network. But when doing quick restarts 13 // and tests, that delay is annoying, so I turn it off here. 14 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 15 16 // then the server needs to bind to an address that other 17 // people can connect to... You do not need to give the full 18 // address unless you only want to listen on one particular 19 // interface. Just the port is OK if you want to be accessible 20 // from any network interface the computer is on. 21 // 22 // note that this can throw an exception if the address is 23 // already in use by another program, or if your user account 24 // doesn't have permission to use the port (ports < 1024 are 25 // traditionally restricted to root on Linux, for example) 26 listener.bind(new InternetAddress(2525)); 27 // then start listening. The number here is how many 28 // connections it will keep waiting instead of rejecting 29 // when it is busy. 30 listener.listen(10); 31 32 accept_another: 33 34 // then the listener socket just accepts connections 35 // normally, at this point, you would switch to event-driven 36 // programming and accept new connections only when they 37 // actually come instead of waiting for one. But for this 38 // sample, we will just wait until someone connects by 39 // calling `accept`. I'll show a simple event sample later. 40 Socket socket = listener.accept(); 41 42 // and now you can communicate with the connection returned 43 // by `accept`, which is just another regular socket of the 44 // same type. 45 string message = "Hello!"; 46 47 // now we have to accept the reality that our message may 48 // be broken up by the network stack. As such we're going 49 // to loop over until the whole thing is sent. 50 while(message.length) { 51 auto ret = socket.send(message[]); 52 if(ret == Socket.ERROR) 53 throw new Exception(lastSocketError()); 54 55 if(ret == 0) 56 return; // the other side closed, we're done 57 58 // advance our message for the next loop so we don't 59 // resend any part of it. 60 message = message[ret .. $]; 61 } 62 63 // when you are done sending, you can call shutdown 64 // to indicate that no more will come. You don't have 65 // to do this though, instead you might just close when 66 // you're done. But since I want to wait for a reply, I will 67 // simply shut down sending so it can still receive before 68 // closing. 69 socket.shutdown(SocketShutdown.SEND); 70 71 // and now we'll wait for the reply. 72 73 string reply; 74 75 do { 76 // messages can be received in arbitrary chunks 77 // too, so I am going to copy it into another 78 // reply variable to make that more convenient. 79 // 80 // you might instead just stream data right through 81 // this buffer for more efficiency, but remember 82 // there's *no guarantee* the whole message will come 83 // in with one call, even if your buffer is huge! 84 char[1024] buffer; 85 auto ret = socket.receive(buffer[]); 86 if(ret == Socket.ERROR) 87 throw new Exception(lastSocketError()); 88 // if receive returns zero, it means the other side 89 // shutdown send or closed the connection. 90 if(ret == 0) 91 break; // other side indicated they are done 92 93 reply ~= buffer[0 .. ret]; 94 } while(true); 95 96 writeln("Got from client: ", reply); 97 98 // we received our connection, sent it a hello, and got 99 // a reply. now we're done, so let's close the connection. 100 101 socket.close(); 102 103 // at this point, we could loop back up and accept another 104 // one; `goto accept_another;` but for this demo I am just 105 // going to call it finished and exit the program. 106 // 107 // closing the listener too is not strictly necessary but 108 // it is nice to close stuff you are done with. 109 listener.close(); 110 }
1 import std.socket, std.stdio; 2 3 void main() { 4 // a client socket can be simpler - it just needs to 5 // create the socket, connect, and then communicate 6 auto socket = new Socket(AddressFamily.INET, SocketType.STREAM); 7 // note that connect can throw an exception if the connection 8 // fails (try it without running the other server program!) 9 socket.connect(new InternetAddress("localhost", 2525)); 10 11 writeln("Client connected!"); 12 13 string serverMessage; 14 15 do { 16 // I am using a char buffer here because I know I 17 // am sending strings on the other side, but you 18 // would very likely want to use ubyte instead for 19 // binary protocols. 20 // 21 // The size of the buffer is also a bit arbitrary, 22 // anything from 1 KB (1024) to 4 KB (4096) are 23 // very common block sizes, but you can do something 24 // else too if you like. Packets come in various 25 // sizes and the kernel buffers the stream until 26 // you receive it too. 27 // 28 // Just it must NOT be zero! Do not declare 29 // `char[] buffer;`, since that has zero length 30 // so it cannot receive anything. 31 char[1024] buffer; 32 33 // receive will return whatever is available now, 34 // which could be more or less than we want, up to 35 // the max size of the buffer. It returns 0 if the 36 // other side has disconnected, or ERROR if something 37 // has gone wrong. 38 auto ret = socket.receive(buffer[]); 39 if(ret == Socket.ERROR) 40 throw new Exception(lastSocketError()); 41 42 if(ret == 0) 43 break; // got it all! 44 45 // now slice the buffer to get the part... 46 auto part = buffer[0 .. ret]; 47 // and append it to build our complete message. 48 // may also be possible to process without copying 49 // in a real program, especially depending on your 50 // network protocol. I like to define one where a 51 // message size is given up front, then you read 52 // a complete message into a buffer and process it 53 // when it is done. Then messages may span across 54 // packets and the end of one packet can be a new 55 // message, but all is handled by just slicing and 56 // copying when necessary. 57 // 58 // But some have other ways to indicate the end, like 59 // a sentinel value you can detect and slice on, or 60 // here, it is just everything available on the stream. 61 serverMessage ~= part; 62 } while(true); 63 64 writeln("Received from server: ", serverMessage); 65 66 // send our reply the same way 67 string reply = "Thanks! Bye."; 68 while(reply.length) { 69 auto got = socket.send(reply); 70 // you might be asking, why doesn't the class methods 71 // just throw? The main reason is these errors might 72 // be handleable, like wouldHaveBlocked(), or EINTR 73 // (see the other example above) and you want those 74 // to be handled here quickly. An exception could 75 // be handled too, but the library preferred to simply 76 // forward the underlying error codes. 77 if(got == Socket.ERROR) 78 throw new Exception(lastSocketError()); 79 if(got == 0) 80 break; 81 82 reply = reply[got .. $]; 83 } 84 85 writeln("reply sent!"); 86 87 // we're done, let's close and exit. 88 socket.close(); 89 }
You can get events for small numbers of active sockets with the Socket.select function. Here's an example of a socket server using that function. You might run several copies of the client above to talk to this. To exit, just press ctrl+c in the terminal running this sample, or connect five clients and it will exit once all five are done.
1 import std.socket, std.stdio; 2 import core.time; // for durations for our timeouts 3 4 void main() { 5 // it starts off the same as the other server example: 6 // create the socket, bind to an address, and start to listen 7 auto listener = new Socket(AddressFamily.INET, SocketType.STREAM); 8 9 // want to reuse again for convenience 10 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 11 12 listener.bind(new InternetAddress("localhost", 2525)); 13 listener.listen(10); 14 15 writeln("listening"); 16 17 // this is a new concept though: a SocketSet holds the 18 // sockets you are asking for events on. It must be reset 19 // each time it is used, but only constructed once, so we 20 // will make it here, then populate it in the event loop below. 21 auto readSet = new SocketSet(); 22 23 // since we can handle multiple clients, let's make an array 24 // for them too. 25 Socket[] connectedClients; 26 bool isRunning = true; 27 28 // I want this program to exit after it handles 5 connections, 29 // so I'll keep count of them here too. 30 int totalConnectionCount = 0; 31 32 // and this is the event loop... 33 while(isRunning) { 34 // Here's how to use that SocketSet: first, reset it 35 // in each loop, then add all the sockets we are 36 // interested in. Afterward, we call `select`, then 37 // loop through again and see if there are still 38 // events in the set. 39 // 40 // So first, reset and add our live sockets: 41 readSet.reset(); 42 foreach(client; connectedClients) 43 readSet.add(client); 44 // including the listener for new connections! 45 readSet.add(listener); 46 47 // Now we call `Socket.select`. It takes arguments 48 // for up to three sets: one for reads, one for 49 // writes, and one for exceptional conditions like 50 // special out-of-band data. 51 // 52 // All three sets work the same way and are 53 // independent. You can pass `null` to ones you 54 // don't care about. 55 // 56 // You can also pass a timeout value if you like, which 57 // i'll do here for demo purposes. The timeout is 58 // triggered if no other events come before the 59 // designated duration. 60 // 61 // BTW the technical definition of "event" with select 62 // is that the given operation - read or write - will 63 // not block when you call the function exactly once. 64 // 65 // A read event is triggered when there's no data if 66 // the other side closes the connection, for example, 67 // because it will return 0 immediately on receive. 68 // 69 // Similarly, an error condition existing on the 70 // socket will also trigger the readiness event since 71 // you can read that immediately. So still check your 72 // return values! 73 // 74 // I am only checking read events, even though you 75 // arguably should check write events too to avoid 76 // being forced to wait for a write buffer to open up. 77 // 78 // Just that complicates the code a lot and if you are 79 // only doing small writes, it is unlikely for it to 80 // keep you waiting as long as reads. 81 auto eventCount = Socket.select(readSet, null, null, 5.seconds); 82 83 if(eventCount == -1) { 84 // select returning -1 is not necessarily 85 // an error, it means the function got 86 // interrupted. For example, a user might 87 // have triggered a ctrl+c interrupt (with 88 // a custom handler, by default that exits). 89 // 90 // since interruption is not fatal, we can 91 // check flags from signal handlers (none 92 // here) and then simply retry. 93 continue; // retry by continuing loop 94 } 95 96 if(eventCount == 0) { 97 // it returns 0 on the timeout. 98 // just print and continue here 99 writeln("Select timed out."); 100 continue; 101 } 102 103 // otherwise, it tells us how many events are 104 // ready. But we don't know which ones without 105 // looping through everything we had added and 106 // testing them. (This is why this API is not 107 // efficient for large numbers of sockets - all 108 // these extra loops add up.) 109 if(eventCount > 0) { 110 // remember to check our listener! if it 111 // is ready, we can call accept on it. 112 if(readSet.isSet(listener)) { 113 // the listener is ready to read, that means 114 // a new client wants to connect. We accept it here. 115 writeln("accepting new socket"); 116 auto newSocket = listener.accept(); 117 newSocket.send("Hello!\n"); // say hello 118 // skipping the error check here for brevity 119 newSocket.shutdown(SocketShutdown.SEND); 120 connectedClients ~= newSocket; // add to our list 121 122 // we handled one event, so let's 123 // note that there's less to do now 124 eventCount--; 125 } 126 127 // I am doing a for loop here rather than 128 // foreach because I will modify the 129 // connectedClients inside and that is not 130 // well-defined with foreach. 131 for(int idx = 0; idx < connectedClients.length; idx++) { 132 // if nothing else to do, no point wasting time looping 133 if(eventCount <= 0) 134 break; 135 auto client = connectedClients[idx]; 136 if(readSet.isSet(client)) { 137 // ready to read from this client! 138 139 eventCount--; 140 141 char[1024] buffer; 142 143 // let's just read from it and 144 // echo it back 145 auto got = client.receive(buffer[]); 146 if(got == Socket.ERROR) 147 throw new Exception(lastSocketError()); 148 if(got == 0) { 149 // they disconnected. let's keep count 150 // and exit if we hit enough 151 totalConnectionCount++; 152 if(totalConnectionCount >= 5) 153 isRunning = false; 154 155 // and close it too. 156 client.close(); 157 158 writeln("socket disconnected"); 159 160 // and then when it is closed, we need to remove 161 // it from our active clients list 162 connectedClients[idx] = connectedClients[$ - 1]; 163 connectedClients = connectedClients[0 .. $-1]; 164 idx--; // modifying the item in the loop is weird 165 } else { 166 // otherwise, we received data and I just 167 // want to echo it back and shutdown. This 168 // would change based on the network protocol 169 client.send(buffer[0 .. got]); 170 client.shutdown(SocketShutdown.SEND); 171 172 writeln("socket sent data"); 173 } 174 } 175 } 176 } 177 178 // after handling these events, we loop back up to 179 // wait for and handle more. Note that select will 180 // wait until it either timeouts or has something 181 // new, so it won't eat your CPU. 182 } 183 }
Reading events from several different sources - like the keyboard or a gui in addition to the network - is something Phobos cannot do. You'll need OS functions or a third party library for that.
Moreover, Phobos' select function is best for small numbers of connections and is not hugely efficient for more (e.g. thousands). For these more complex event models, use operating system functions through the core.sys package, or third party libraries.
In blocking mode, a call to send or receive will wait until a timeout occurs or the data is finished to return. You might want to set these timeouts to a smaller value to make interrupting easier.
To set a send timeout, call
// 3 second example socket.setOption(SocketOptionLevel.SOCKET, SocketOption.SNDTIMEO, dur!"seconds"(3));
For a receive timeout, use:
// 3 second example socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3));
On newest Phobos versions (it was buggy until recently) the wouldHaveBlocked() function will return true after the send/receive call returns Socket.ERROR to find this condition.
I sometimes use smaller timeouts to give a loop a chance to check a shouldExit flag, so we can cleanly exit more easily.
Unblocking mode, setting Socket.blocking to false, is basically like setting a timeout to 0.
Phobos' Socket class has a destructor that closes the socket! If you use the socket handle elsewhere, don't let the Socket get away from you. When it gets garbage collected, it will close the connection... even if you kept a copy of the underlying OS handle somewhere else. This can lead to frustrating random bugs!
If you are going to break the phobos Socket abstraction at all, I recommend never using it and instead use the underlying functions yourself. Thankfully, since std.socket is mostly just a thin wrapper around those underlying functions, your knowledge will almost entirely carry over to using them. Biggest downside is the Windows and Posix socket APIs are slightly different - 80% of your code will probably compile either way, but the rest of it might need some version conditions.
TCP is a fairly complex protocol and there are a lot of ways to tweak your program to get better performance. Socket.setOption can set these tweaks, but the conditions when you should use them and what specifically you should do is out of the scope of this tutorial. But remember, knowledge from C and other languages about their socket APIs will translate to D pretty easily.
Socket tutorial today, it may be simple for experienced programmers but hopefully useful to people new to the idea.