Chat not working.

Fire

@catreina repeating welcome messages is indicative of the chat server rebooting, the duplicate messages from people are indicative of joining the same channel multiple times at once (This happened to me when I accidentally had the same channel favorited in mIRC and had both set to auto-join)

Now, the issues we are all having (and will continue to have) are because the servers they use for chat are extremely unstable. I do not know why they dont just switch to using a real IRC network and then embedding a web chat interface to use said IRC servers, similar to what own3d does.... or, I dunno, use a real IRC server for their own network.  It isnt hard to use something like the unreal daemon or undernet daemon with configuration changes and tweaks to use their own databases....

own3d does not have their own chat servers, they just piggyback off of quakenet with custom channel names. They have absolutely no site integration, and they barely hold a candle to the amount of traffic we have going here. Justin/Twitch runs the largest IRC network on the planet, and basic IRC servers would die in an instant from the load (not to mention the lack of integration with the site).

A custom solution is necessary (The current one being written in Python with Twisted), and they have been re-writing a new extremely high performance chat server from scratch over the past 6-8+ months due to the instability the current server network was showing (which has been ever-increasing). The current constant reboots seem to be caused by moving channels from the current chat servers to the beta chat servers (there's an opt-in somewhere, I don't know where right off hand, it's features are limited)

@everyone I understand all of your anger, but keep in mind they're working their asses off trying to get this fixed. It's not fun for any of us, and if the hate train leaves the station it's going to end badly for all aboard.

I, @havokk00 , and the rest of the mod team are doing our best to keep you informed so you understand what is going on and aren't just left out in the cold. I realize that's not as good as a fix, but it's the best we can do considering the circumstances. 

catreina

The duplicated welcome messages and from people is NOT indicative of what you are claiming - when I get reconnected (with the information I posted) the duplicated messages are received AT THE SAME SECOND, IN THE SAME WINDOW, FROM THE SAME PERSON - and it is consistent no matter who it is from.  That said, I am not a friggin moron - I have been using IRC both as an IRCOP and IRC ADMIN for over 15 years. You trying to tell me that my duplicated messages issue is somehow because I am unknowingly connecting to the same server more than once with one instance of mIRC (or any other chat client) seems to again be you and the Twitch staff attempting to place blame on the users/casters instead of squarely where it belongs - on the servers that twitch / justin.tv runs and provides.

own3d uses Quakenet - while at the same time using a java web interface for it.  That said, your claim that justin/twitch runs the largest IRC network on the planet is both flawed and disingenious - it is not a network when you cannot A) join ANY channel by connecting to ANY server (hence the network part) and B) it is not the largest by any means - Undernet Efnet and others I can practically guarantee still have your fake network beat in terms of size.

Additionally - as I said in my post, using a current daemon with massive tweaks to the code would have been the best choice, even when creating one from scratch. your people having been coding this for 6-8 months shows that you are not doing it right.

catreina

Let me give you an example of how flawed your "if they used current IRC daemons" argument is

World of Warcraft uses custom chat servers - arguably without the functionality, but bear with me here - They restart the chat servers when they restart the game servers, once a week.  There are never problems with chat, and I would venture a guess that the chat in WoW is at least as heavily used as some of the channel groups in Twitch - even more so with all the addon messages using their own channels.

once again, I have to reiterate - the servers Twitch uses for their "network" is hardly "the largest on the planet", as per my prior post.

zerathi

Fire, really. I'm not taking anyone's side because I'm not one to ride the bandwagon, but it isn't the regular daily chat maintenance reboot. The welcome message just keeps repeating, over and over. You're right, though. The chat system is extremely unstable in many ways, and now it seems to be getting worse. I hope they can fix the problems very soon as it's not only an inconvenience for the streamers, but obviously for the viewers. And, we all know what happens when viewers are inconvenienced. They leave, and they leave quick. 

catreina

There is a reason IRC (Quakenet) has survived for as long as it has - it is simple, easy to use, and it has massive redundancy - in the event of a server dying, on most server networks (Quakenet) you reconnect instantly to another leaf (server) in the network, and then back to the channel you were in because of the redundancy. 

If it aint broke, dont fix it.... you will break it doing so (Twitch/JTV)

Fire

@catreina Did I say connect to the same server more than once? No, I said joining the same channel more than once. You only have to be connected once, but you can send multiple join commands for the same channel.

It's a pseudo-network sure (because you have to connect with your jtv un/pw to a specific IP or subdomain of jtvirc), but as for the largest? We will have upwards of 200,000+ users in a single channel during events. A single channel. Then you add the other popular streams with thousands and thousands of users in those chat rooms, across the entire website? Being the largest IRC network is not hard to see.

World of warcraft's chat isn't really an accurate comparison, they likely do not in any way use IRC as their base, they don't have to conform to specifications, and can build it specifically to their needs.

As for tweaking a current daemon instead of writing their own, I don't know the details of what exactly they are building, I do know though that it's extremely fast and very stable, even in it's early stages. All the channels seem to be under a single umbrella in the current iteration like a standard server as well.

If you want to continue bashing the site and it's infrastructure I suggest you take it up with the engineers themselves. I've given you what I can and will not continue a pointless argument about things neither of us have control over.

@zerathi yeah, that's what I was saying, it's not the standard reboots, something's causing the network to dump at weird times.

moocat

The chat servers being shitty is nothing new, its been going on for years. The problem is they've now spent, what, a year since the announcement of a new chat backend, and the beta (alpha) chat is barely even barebones. Not sure what they are doing all day, but hopefully they will get to it soon.

I'm glad though that they don't go the easy route and just embed some shitty Quakenet chat like own3d does. The chat on this site might be unstable at times but when it works its a huge difference to the service.

catreina

@Fire - 200,000 users in a single channel has been done before on other networks. Additionally, the users in other channels are usually people in MULTIPLE channels at once.  Largest network again is a misnomer - how many chat servers does JTV have?  How many does undernet have? And how many peak users did undernet boast at its peak?  

I do admit the WoW comparison was quite a bit off, but it was for making a point only.

It is very frustrating to have been dealing with the double chat, the issues with the instability of the servers, the recent issues with the change in revenue statistics/metrics, the constant lack of communication on all fronts, and then having someone telling me A is B - well, like I said, it is very frustrating. Not bashing the site (making suggestions in a volatile manner, but not outright bashing JTV/Twitch)

I also thought you were an employee of Twitch - I was going to bring some of this stuff up to Kevin when I met him at Pax East, but he was too busy and I didnt want to intrude directly about something he had no control over -at that time-

Fire

@catreina At the time of this posting we have 9 physical machines in the cluster, with ~140k active users total (~15.5k per machine). There are also no events going on to my knowledge, which reduces the count drastically (IPL peaked at 240k unique users in just that channel).

And that point was an inaccurate one to make, they're hardcoded for access from their servers only, everything is very very tightly controlled and optimized (and then re-optimized when the addon channel spam became such a huge problem).

I think I understand what you were trying to get across though (the fact that they are only rebooted once a week). If it's any consolation, I've been connected to the beta chat's server for ~10 days and there hasn't been a single reboot (I think features have been added during this time as well).

The revenue thing was a bit odd I agree, but you should have been able to email your parnership handler or partnerhelp and get a quick answer about it. I think I also answered it myself here on the forums a time or two.

I understand the frustration, not only am I a streamer myself, but I also moderate for other large channels. When issues with the servers crop up all I can do is groan, facepalm, and wait for it to (hopefully...) fix itself.

And lastly nah, not Staff (that's the dream/end goal). I'm just a support moderator, and a proxy/buffer between them and the hostility/irrationality/attacks users so easily (and frequently!) pour into this forum :P

magnetssc

Chat absolutely refuses to load for me on my stream.  What I have noticed is that it does load for all other streams I go to, worst case scenario a 20 second rotating circle showing that it's loading.  However, directly to the channel or through the dashboard the chat sits with the rotating circle permanently.  Has sat like that for 6 hours in chrome, I've attempted loading in firefox, IE 32 and 64 bit as well as logging off and even restarting the system, same exact issue no matter what.

magnificentaurora

People are getting sick of this and most of the time I'm hearing it's on Twitch. My chat intro message loads over and over and over again. And I don't wanna hear that the 'chat is rebooting' it's frickin unusual for it to do it THIS much! Also, some of my viewers messages repeat themselves when they only type a particular message once! Another broadcaster/friend says some of his viewers' message don't load unless he continuously refreshes his chat. This keeps up I'm going to consider stopping twitch because I can't take this....

cyanidecapsule

From what I've seen, people with beta chat are having no problems except not being able to time-out viewers and name color issues. People without beta chat can't have a sensible conversation with their viewers...I've been in several of my favorite channels where the caster had to stop casting because of this issue -.-

switchsl

I am using twitch app on an iPad3 and am having the same problems with chat. I just updated the app, and am now not even seeing a chat icon to let me into chat. Also I cannot minimize the window after maxing it. The minimize just closes out the stream entirely. Twitch is long known for chat issues and incredible lag issues when hosting big live events, and it never seems to get better. It's not like I can do anything about it, but I am just airing thousands of other people's thoughts. Own3d uses seperate irc chat and has less than half the problems. Something to think about, and saying twitch has more viewers is laughable.

jsauce

All other issues aside, this problem comes down to one thing. This is a chat server, where people of like minded views come to converse about said like minded views, in most cases. You come to watch a stream and chat with people about what you are watching most of the time, and sometimes you stray a little and talk about off-topic things as well. In any case, the point of the chat server is to chat. The problem with the chat server is YOU CAN'T CHAT. You can't have a normal conversation because you are frequently disconnected mid sentence, every 2 to 3 minutes without fail. And then on top of that you must wait sometimes 3 to 5 minutes to get reconnected. I have personally been disconnected and had to wait more than 45 minutes to get connected again, only to be disconnected again with a few minutes. It's become pretty apparent to most of us that the people in charge have thrown up their hands and just given up, this is indicated by no attempt at rectifying the situation. Why do I say this? Because the problem still exists. Let me tell you something, twitch/justin tv are businesses. In the business world, having worked in it, I can tell you that such a problem would have been rectified in a matter of minutes, hours at the most. And if not, I guarantee some people would have been fired, and new people would have been brought in immediately to fix the problem. Being that this issue has persisted now for days with no end in sight, my only guesses are 1) Either twitch/justin tv does not care about its customer base. 2) Twitch/justin tv are incapable of running a network like this and people must search out a new way of streaming/chatting or 3) Twitch/justin tv is actually run by a pack of Chimpanzees who run the company like you would expect wild animals to run a zoo.

Although I must admit, I'm leaning toward it being the third option, option 1 and 2 seem possible. If Twitch/justin tv are incapable of rectifying the situation, it seems evident that most people will leave for an alternative. Factor in the attention span of your average teenager and what is the likelihood that this whole thing blows up in the faces of the people in charge of this company if they do not see fit to fix it immediately. Go a step further, because what if the people who stream the content instead take their business elsewhere? Those teenagers who get their content here will easily go where it is found, and certainly where it is less problematic to do so. And finally if the first option is true, then the result of the second seems also to be in their future. Companies are built on the revenue of the consumers who purchase their product. People come here to see a streaming video, pay a subscription fee in some cases, and in other cases watch ads. In either case this is a revenue stream for Twitch/justin tv, and to not care about customer satisfaction goes against the general principle of business, that each customer matters and each customer is always right. Your customers have spoken, we want a solution, not tomorrow, not next week, not in six months, not next year, we want a solution now. 

theundeadfairy

I am really upset because a chat system is supposed to work where you type something and you would expect it to be displayed to the other people in chat. It is really frustrating also to type something into chat box enter, goes into chat window and then a few seconds later it dissapears. I will not use twitch if it is not reliable. It is happening to everyone else. Do you really want to be known for having problems with chat???? 

cirnotv

Twitch made a comment on Twitter a few days ago saying chat issues have been "resolved" this is obviously not the case. Everywhere I go, from small streams to big streams the chat just does not work. People are constantly "losing" their messages, and I'm constantly needing to re-type things because I have no clue if they made it through or not, that coupled with all the frequent disconnecting makes something that is supposed to be fun and relaxing a huge nightmare since having a conversation is impossible. Is the problem really so severe it can't be fixed right away? or is nothing just being done about it? Either way, as a person who likes streaming, and as a person who also enjoys watching many other streams, having a non functioning chat server is like having a ferrari with no wheels. Sure, it may look nice and be presented well, but what the shit are you going to do with a car that has no wheels? Big stream or small, this issue affects everyone. I think the only reason a lot of people haven't switched over to another streaming website is moving your stream and fan base over could prove to be quite a big pain. So please, please fix this very soon or else a lot of streamers and users may have to find another alternative.

duosigma

well, my chat constantly spams my welcoming message, and i cant see any of my viewers messages, this isnt borthersome at aaaalll..

cvagts

I understand that the chat server would have problems under this kind of stress, and I assume that TwitchTV is working on it, as has been said in this topic.  My problem is that I have seen almost ZERO from TwitchTV support announcing to everyone, "Sorry guys, here's what is wrong with our chat servers, here is what we're doing to stop it, and if there's a timetable on when it will be fixed, here it is.  We apologize for the inconvenience, and we assure you that it will be better than ever as soon as we can finish maintenance."  I haven't seen ANYTHING, other than a tweet or two in passing, as well as posts in this topic, and neither of those have reached the masses of Twitch.  If there have been messages, I apologize for having missed them, but I really think there should be more of a concerted effort to tell people what's going wrong and when they can expect things to be fixed.

morph4037

Yea this is getting pretty old. Been going on for the last two weeks+ and it's only getting worse. I frequently watch shift_tv's channel and chat is almost always broken. Server restarting, randomly breaking, and just not working in general. Dude is getting frustrated and I don't blame him at all. Also wouldn't be surprised if more and more streamers are starting to feel the same way. 

itzrokk

Google Chrome, chat will suddenly stop streaming down. Have to reload. Multiple times on multiple streams.

xpdxtv

This issue has been going on for weeks now it seems. Chat seems to work very finicky at best, everyone in the chat seems to be DC'd every 5-30 minutes and the chat either re-loads or doesn't load at all. As a channel streamer i find this very frustrating not being able to interact with my crowd. Can we please have a legitimate explanation atleast as to why chat is barely working and sometimes not working at all? Or even better can some sort of fix be posted that is an actual fix. The twitter update said these issues were fixed but obviously they haven't as people are all over are still having issues. Please don't take this as "rage" all i want is to have my chat working again so i can interact with my viewers.

icedoutswag

I'm having the same problem. Half the time my chat won't load for the viewers and when it does it keeps restarting (gives them my welcome message after it freezes for a couple minutes) and a lot of the time what my viewers are typing doesn't actually show up in the chat. Any suggestions or help? Thanks

thecommadore

Twitch IPad app chat does not work. It will not load

pcbigboy

From a broadcaster POV who uses an IRC client to access the chat room, my problem AT FIRST was joining the room.  I could successfully connect to the server but that's it.  THIS HAS SINCE BEEN FIXED !!!

My problem NOW has been STAYING connected to the server.  Not sure if it's due to timeout associated with idleness (Which of course is caused by no one being able to access the room from the website) or what it is.  In either case, while the chat problems aren't **DIRECTLY** tied to Twitch TV, we're suffering just the same because we are on the same servers as JTV broadcasters

Perhaps JTV/Twitch needs to consider putting Twitch TV chat rooms on a SEPERATE chat room network (With JTV servers being used as backend backups) from those of JTV users (With Twitch TV servers being used as backend backups for them) ??  Just a thought.....

Cheers :)

Pat (AKA pcbigboy)

runprogramgaming

i would just like to thank @Fire for replying so often to people having issues on the forums. Must be a lot of work answering all of the questions constantly and the fact that you are not staff and do it is pretty cool. Also, i seriously think if twitch is going to have such a high update that affects everyone channel like this then they should just shut the channel down for a few days, get it fixed and be done with it. Updating and making changes while the site is up just causes issues for everyone.

xylax

Something is getting worse and worse day by day lately. .It's not even that it's not loading - it does, but no messages appear, they get doubled, or it lags, or suddenly stops refreshing. It's getting really bad in last 2-3 days and continues on being bugged even more by every hour

chaotsc

the chat is super laggy (takes 5-10 seconds for chat commands and msgs) and i get timeouts from the chat server every few minutes which results in getting spammed with the welcome message over and over again in my own channel... can't talk to viewers via chat like this and am unsure if i see anything they write while watching my stream... i tested this on 2 different pc's with chrome and chromium, plus a friend of mine is having the same problem (also using chrome). i'm having this issue for like a week now... did you guys move your ircd to a single server at the northpole or something? sad story :/

chaotsc

so i just fired up irssi on one of my servers to see if that helps with the chat issue... (100mbit box, different isp/routing than my 60mbit homeline)

ns lookups for <channel>.jtvirc.com (checked 8 different channels) gave me just 2 ips to connect to.

icmp packets & standard traceroutes are blocked on both, tcp trace on the ircd port gave me 6 hops both at 160.000 ms at the last hop.

so at first glance it didn't look that bad latency wise, however once connected to either of the ips lag goes up to 30 seconds and disconnects occur pretty much exactly in 5 minute intervals.

increasing the timeout settings in irssi didn't do anything either.

looks like the problem isn't browser related and since the route doesn't seem to be an issue either i'd wildy guess that it's not network related at all but some ircd software or hardware issue on the jtv side due to overload (or maybe the isp just started dropping connections every 5 mintues for the giggles? :P )

whatever the the problem actually is, i hope it gets fixed as soon as possible and communication goes back to bidirectional :D

so thats my 5 cents on the topic...

cheers

bgeorge

@chaotsc - Yes, its basically the chat server/ircd failing on Twitch's end.

xupyourgamex

I'm having a similar issue.  The chat continually refreshes/reloads or just freezes.  I'm also getting wrong viewer count numbers as well.  

http://www.twitchtv.com/XUpYourGameX