SteamRE - Reversing the Steam protocol

[hakusaro]

Fun, C# is so deadly close to Java, porting won't be hard at all really, just meticulous. It'd be fun to see this running on Java, for things like GoogleTV(!), Android, etc.

[ZeekyHBomb]

I'm playing with logging into Steam3 and I have a question about SteanClient Callbacks.
Right now a polling system is used, but one could easily make use of events and just invoking it instead of pushing them on a queue in PostCallback.
If that seems bad due to multi-threading (considering the queue-access is protected by a lock) you could simply use a System.Threading.AutoResetEvent and Set it in PostCallback.

[yngndrw]

Oops missed this thread. Very nice work.

I haven't forgotten about my Web-Based-Steam-Chat project, just haven't gotten around to it yet. This library would work well with ASP.Net for it.

Does the Steam protocol still have support for those old Steam games there used to be - I wonder if they could make a come-back with custom clients ?

[editline]11th February 2011[/editline]

[QUOTE=nicatronTg;27985015]Android[/QUOTE]
[url]http://tirania.org/blog/archive/2011/Jan-04.html[/url]

[Blade]

[QUOTE=ZeekyHBomb;27991656]I'm playing with logging into Steam3 and I have a question about SteanClient Callbacks.
Right now a polling system is used, but one could easily make use of events and just invoking it instead of pushing them on a queue in PostCallback.
If that seems bad due to multi-threading (considering the queue-access is protected by a lock) you could simply use a System.Threading.AutoResetEvent and Set it in PostCallback.[/QUOTE]

I just noticed that, will look into changing it around.

[ZeekyHBomb]

[QUOTE=high;27998929]I just noticed that, will look into changing it around.[/QUOTE]

My proposal:

Index: SteamKit2/SteamKit2/Steam3/SteamClient/SteamClient.cs
===================================================================
--- SteamKit2/SteamKit2/Steam3/SteamClient/SteamClient.cs	(revision 172)
+++ SteamKit2/SteamKit2/Steam3/SteamClient/SteamClient.cs	(working copy)
@@ -9,6 +9,7 @@
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
+using System.Threading;
 
 namespace SteamKit2
 {
@@ -22,8 +23,9 @@
 
         object callbackLock = new object();
         Queue<CallbackMsg> callbackQueue;
+		ManualResetEvent callbackAvailable = new ManualResetEvent(false);
 
-
+		
         /// <summary>
         /// Initializes a new instance of the <see cref="SteamClient"/> class.
         /// </summary>
@@ -83,6 +85,41 @@
         }
 
 
+		/// <summary>
+		/// Gets the next callback object in the queue, or if none is available blocks until one is. This function does not dequeue the callback, you must call FreeLastCallback after processing it.
+		/// </summary>
+		/// <returns>The next callback in the queue. Will never return null.</returns>
+		public CallbackMsg WaitForCallback()
+		{
+			CallbackMsg msg;
+			
+			do
+				callbackAvailable.WaitOne();
+			while( ( msg = GetCallback() ) == null);
+			
+			return msg;
+		}
+		
+		/// <summary>
+		/// Gets the next callback object in the queue, or if none is available blocks until one is. This function does not dequeue the callback, you must call FreeLastCallback after processing it.
+		/// </summary>
+		/// <param name="timeout">A TimeSpan that represents the number of milliseconds to wait, or a TimeSpan that represents -1 milliseconds to wait indefinitely.</param>
+		/// <returns>The next callback in the queue. Might return null.</returns>
+		public CallbackMsg WaitForCallback( TimeSpan timeout )
+		{
+			return callbackAvailable.WaitOne( timeout ) ? GetCallback() : null;
+		}
+		
+		/// <summary>
+		/// Gets the next callback object in the queue, or if none is available blocks until one is. This function does not dequeue the callback, you must call FreeLastCallback after processing it.
+		/// </summary>
+		/// <param name="millisecondsTimeout">The number of milliseconds to wait, or Timeout.Infinite (-1) to wait indefinitely.</param>
+		/// <returns>The next callback in the queue. Might return null.</returns>
+		public CallbackMsg WaitForCallback(int millisecondsTimeout)
+		{
+			return callbackAvailable.WaitOne( millisecondsTimeout ) ? GetCallback() : null;
+		}
+		
         /// <summary>
         /// Gets the next callback object in the queue. This function does not dequeue the callback, you must call FreeLastCallback after processing it.
         /// </summary>
@@ -106,7 +143,9 @@
         {
             lock ( callbackLock )
             {
-                if ( callbackQueue.Count == 0 )
+				if ( callbackQueue.Count == 1 )
+					callbackAvailable.Reset();
+                else if ( callbackQueue.Count == 0 )
                     return;
 
                 callbackQueue.Dequeue();
@@ -125,6 +164,8 @@
             {
                 callbackQueue.Enqueue( msg );
             }
+			
+			callbackAvailable.Set();
         }
 
 

One thing to think about is what should happen if we wait for another callback (right now it'll block indefinitely), but none comes in, though the user could take care of that by setting a timeout, provided via two overloads.
If there is some kind of exit condition that verifies that no callback will ever come in (does the handler remove itself when the internet connection gets lost?) one could just use another ManualResetEvent and instead of callbackAvailable.WaitOne(); use WaitHandle.WaitAny( { callbackAvailable, lastHandlerRemoved } ); and return null if it returns 1.

Another thing I've noticed are unnecessary references in the SteamKit2 project, which are System.Core, System.Data and System.XML.
Further more did it compile fine for me when targeting .NET 2.0; .NET 1.1 didn't work because apparently the Visual Studio 2008 project-file is not compatible with that.
I'm by no means a C# or .NET whiz, but wouldn't it be better to set the target as low as possible to make it run on out-dated systems as well? Are there any benefits when targeting higher versions if you don't use any newly introduced features?

[editline]12th February 2011[/editline]

Oh, and yet another small question, since one will most likely call FreeLastCallback after GetCallback, would it not be sound to add a boolean-parameter to GetCallback that decides whether or not to automatically dequeue the callback?

[VoiDeD]

The main issue with blocking for callbacks is the fact that monitoring for callbacks should be done in the UI thread. This way there are no invoke calls or other cross-thread calls being done.

As for the references to System.Core, SteamKit2's lowest .NET target is 3.5. There are various places in the code were we make use of the features provided by that version.

[QUOTE=ZeekyHBomb;28003564]
If there is some kind of exit condition that verifies that no callback will ever come in (does the handler remove itself when the internet connection gets lost?)
[/QUOTE]

As long as a SteamClient instance exists, callbacks can potentially be posted.

[QUOTE=ZeekyHBomb;28003564]
Oh, and yet another small question, since one will most likely call FreeLastCallback after GetCallback, would it not be sound to add a boolean-parameter to GetCallback that decides whether or not to automatically dequeue the callback?[/QUOTE]

Sounds good, I'll add that next chance I get.

As for some insight into where this whole design comes from, I borrowed the idea of callback polling from Valve's actual steamclient API. I've always been fond of it, so I implemented a similar thing with SteamKit.

Generally their system of polling callbacks is used for games inside the main game loop, but the same can still be applied for windows applications with the WndProc.

The only downside to this design is in WinForm apps in that emulating a WndProc-like update loop isn't very pretty, as shown in the Vapor example: [url]http://svn.limetech.org/repos/steamre/Projects/Vapor/Vapor/Program.cs[/url]

[ZeekyHBomb]

[QUOTE=VoiDeD;28003616]The main issue with blocking for callbacks is the fact that monitoring for callbacks should be done in the UI thread. This way there are no invoke calls or other cross-thread calls being done.[/QUOTE]

I think this should be in the hands of the developer using the library, not the one developing it.
Both methods, polling or waiting, would be kept in tact.

Instead of providing the WaitForCallback methods you could also just implement the ManualResetEvent and provide access to it as a WaitHandle (so you can't Set or Reset from the outside).
The boilerplate code to wait for this handle and then get a callback is really minimal and the design quite dynamic, since you can also choose to block until other handles are signaled, block just a certain time or not at all (WaitOne(0)), though if this is the case one would probably just directly call GetCallback and check for null, although it might be a little cheaper since you don't have to lock the queue.

[Blade]

Is there a reason you used a callback list over an event? Any reason you don't have a dispatch thread inside SteamClient that handles the callbacks instead of the Update loop?

[VoiDeD]

[QUOTE=high;28006398]Is there a reason you used a callback list over an event? Any reason you don't have a dispatch thread inside SteamClient that handles the callbacks instead of the Update loop?[/QUOTE]

Because user code can implement that, if that's the type of functionality they require.

[editline]12th February 2011[/editline]

I am, however, taking Zeeky's advice and I am implementing the WaitForCallback functions.

[Blade]

[QUOTE=VoiDeD;28012974]Because user code can implement that, if that's the type of functionality they require.

[editline]12th February 2011[/editline]

I am, however, taking Zeeky's advice and I am implementing the WaitForCallback functions.[/QUOTE]

What other functionality would someone want 0,o. And if they do want something different couldn't they just create a new class that inherits SteamClient?

[VoiDeD]

I figured I'd update this thread with some of the latest happenings, so here goes.

[url=http://www.facepunch.com/threads/1057523-Vapor-Cross-Platform-Steam-Client]Vapor[/url] is still being developed and has a few new features compared to it's first release.

The major update I wanted to mention is that SteamKit2 now has the functionality to connect to content servers and download game depots. Along with this, I've committed a new proof of concept project called DepotDownloader. You can view it here: [url]http://tracker.limetech.org/projects/steamre/repository/show/Projects/DepotDownloader/DepotDownloader[/url]

This program essentially allows you to download game depots similar to how hldsupdatetool downloads the files required to deploy source engine servers.

All in all it's just another step forward in giving developers more opportunities to interact with the Steam network.

One thing, however, that I've been curious about is if anyone is working on a port of SteamKit2 to another language. Be it Java or C/C++, there's a lot of potential out there to move SteamKit2 to another language, and I know a lot of people would want to see a steam client for their mobile platform of choice, so I wonder if anyone has attempted this.

It would be a pretty massive undertaking, since the networking being Steam is very complex and has a lot of intricate details, but I'm curious nonetheless.

[s0beit]

Good stuff.

I can convert it to C++, if you want, when i have some time. I'm on IRC.

[Colton Rappe]

[QUOTE=s0beit;28694387]Good stuff.

I can convert it to C++, if you want, when i have some time. I'm on IRC.[/QUOTE]

I am all aboard with this idea, I could use the C++ for a learning curve for my C++ Lessons.

[Mattk50]

i would love it if someone here wanted to make a steam trillian plug in

[editline]4th April 2011[/editline]

and so would the few thousand people who want one

[Sykronic]

Awesome
now the linux community will develop a steam client that can't play games.
but thats enough for me!

[VoiDeD]

Small SteamRE update, but the project's main codebase has moved over to bitbucket.

You can now find it here: [url]https://bitbucket.org/VoiDeD/steamre[/url]

Hopefully this will make it easier for other developers to contribute changes, patches, etc.

[Jookia]

Does this violate the Steam Subscriber Agreement?

[url]http://store.steampowered.com/subscriber_agreement/[/url]

Section E, for example.

[deloc]

[QUOTE=Jookia;30213709]Does this violate the Steam Subscriber Agreement?

[url]http://store.steampowered.com/subscriber_agreement/[/url]

Section E, for example.[/QUOTE]
okay.

[ThatLuaCoder]

[QUOTE=Jookia;30213709]Does this violate the Steam Subscriber Agreement?[/QUOTE]

"emulate or redirect the communication protocols used by Valve in any network feature of the Software, through protocol emulation, tunneling, modifying or adding components to the Software"

Yeah.

[TehWhale]

[QUOTE=ThatLuaCoder;30231528]"emulate or redirect the communication protocols used by Valve in any network feature of the Software, through protocol emulation, tunneling, modifying or adding components to the Software"

Yeah.[/QUOTE]you didn't read the whole thing

[quote] (ii) host or provide matchmaking services for the Software or emulate or redirect the communication protocols used by Valve in any network feature of the Software, through protocol emulation, tunneling, modifying or adding components to the Software, use of a utility program or any other techniques now known or hereafter developed, for any purpose including, but not limited to network play over the Internet, network play utilizing commercial or non-commercial gaming networks or as part of content aggregation networks, without the prior written consent of Valve;[/quote]

[blankthemuffin]

[QUOTE=TehWhale;30231609]you didn't read the whole thing[/QUOTE]

You didn't read the whole thing.

"including, but not limited to"

[deloc]

in case no one realized, none of us really give a shit about the ssa. the fact that the project even exists should be an indicator of that.

[VoiDeD]

I'd just like to mention that we do care about the SSA. I'm an avid user of Steam myself, as are the members of our team, and we would rather stay on good terms with Valve.

That being said, it is indeed true that this project goes against the agreement.

However, it is my belief that this project aims to do more good than harm, and exists solely for the purpose of providing information and insight into the inner workings of Steam, along side with enabling interoperability with others who share the same ideals as us.

[foszor]

[QUOTE=VoiDeD;30234182]That being said, it is indeed true that this project goes against the agreement.
[/QUOTE]

Aww man, we were working on an Android chat client. I don't want to see all the time and work become worthless when Valve shuts it down.

[dajoh]

[QUOTE=foszor;30263277]Aww man, we were working on an Android chat client. I don't want to see all the time and work become worthless when Valve shuts it down.[/QUOTE]

I doubt they would do that.

[asherkin]

They would if you tried to monetize it, but I think any free projects should be fine.
If you have something to show off yet, it's worth contacting them.

[foszor]

[QUOTE=asherkin;30263430]They would if you tried to monetize it, but I think any free projects should be fine.
If you have something to show off yet, it's worth contacting them.[/QUOTE]

I made an Android app for viewing Cyanide & Happiness... I didn't have ads and I didn't charge for it... they shut me down via a legal letter to Google from their lawyer.

I did try contacting them, waiting for a response.

[Gook Nukem]

I'm pretty sure Valve wouldn't care unless you were trying to make money or something. It would be best to contact them and ask permission first, they might even set up a deal with you like they did for Garry. :smile:

[gparent]

[QUOTE=foszor;30360492]I made an Android app for viewing Cyanide & Happiness... I didn't have ads and I didn't charge for it... they shut me down via a legal letter to Google from their lawyer.

I did try contacting them, waiting for a response.[/QUOTE]
They want to monetize it I bet... Kind of a douche move not to contact you first.

[CawldFussuian]

You can probably sell it, And get away with it, But you should probably just send somebody at valve an email and ask what they think about it.