Please test your releases better [Archive]

View Full Version : Please test your releases better

alloowishus

November 24th, 2024, 19:04

Smiteworks REALLY needs to test their releases more. Being a software developer myself, there are always a few glitches during a release, but the shear number and severity of the bugs in this release for something I am paying for is quite unnacceptable. PLEASE, please, test your releases better, if they come out slower, that's fine, I am sure we would all be happier with waiting a little longer for something that is more stable.

Thank you.

Sedart

November 24th, 2024, 19:19

I agree, your words make a lot of sense.

HywelPhillips

November 24th, 2024, 20:14

One side note to the OP - unless you are on an ongoing subscription, you're probably not actually paying for these releases? I know I'm certainly not. I paid a one-off payment to buy FGU four years ago and I haven't paid a penny for the core product since. I've been buying plenty of content through the store, which I assume goes towards paying for the ongoing development efforts. But I'm not actually paying for it per se (a la Adobe subscriptions).

This release has been unusually buggy, I agree.

I didn't have the time to help test it, it was in the beta release phase for weeks and presumably these issues just didn't get caught. I know it helps the team a lot if more of us can test - especially in terms of making sure the code is tested in a wide range of deployments and configuration. I try to pitch in when I have time, but I've just been too busy to help with the last couple of releases. But clearly the community only has a certain amount of capacity to test in advance of release too.

So I do wonder whether there's anyone on the team who could set up a more rigorous testing regime? Especially since many of the bugs this time have been "X has stopped working or massively slowed down". It's not that the new features were introducing bugs - core functionality like the Asset window broke when doing refactoring to make the code multi-threaded. This is the sort of thing I imagine a decent regression testing suite ought to be able to find, but what experience I've got of setting that sort of thing up is 20 years out of date and in a very different programming context. I wonder if someone cleverer than me can come up with a decent testing suite?

I believe a bunch of the extension developers have been calling for a less ad hoc testing approach, too. A lot of the refactoring work going on feels like it needs more unit testing before release, again pointing to a need for more automated test suite tools for the devs and less relying on lots of eyeballs at the beta test stage?

Cheers, Hywel

Zacchaeus

November 24th, 2024, 22:07

I think, probably, most of the testing would not have been done in a real time environment. That is people wouldn't have been playing actual games using the test channel. It's also difficult to do a testing with a variety of systems and set ups and with the kind of images people use.

But maybe it would be possible to set up a beta test campaign with a number of Smiteworks chaps and run that for a few hours. Certainly this release seems to have caused more problems than usual.

HywelPhillips

November 24th, 2024, 22:15

Very good points, Zacchaeus. Especially the "actually running a live game" one, even when I have had at go at testing beta releases for FGU the most I've done is play with new features with only myself as the GM connected. A policy of running an hour or two's sessions live certainly might help catch big performance issues.

Maybe an advertised multiplayer "Death Match" arena fight session with pregen PCs, a few monsters and some by-design system-stressing maps for a few beta testers to try?

Cheers, Hywel

LordEntrails

November 24th, 2024, 23:15

Since major releases are available in the beta channel for sometime before deploying to live, would there be a group of folks willing to run one-shots in the test channel prior to go lives?

SilentRuin

November 25th, 2024, 01:29

dradams

November 25th, 2024, 02:45

fantasylord

November 25th, 2024, 03:07

portential

November 25th, 2024, 06:44

Not sure what the release date was for this release, but my two cents are:

-Do releases Monday morning at the start of everyone's work day at Smiteworks
-Start dealing with the bugs with the goal of having FG playable by Friday
OR
the days when FG has the highest use, I'm assuming Fri-Sun would be the highest use.

RosenMcStern

November 25th, 2024, 10:01

This is the sort of thing I imagine a decent regression testing suite ought to be able to find, but what experience I've got of setting that sort of thing up is 20 years out of date and in a very different programming context. I wonder if someone cleverer than me can come up with a decent testing suite?

The customers have no responsibility in whether a release is buggy or not. The TEST channel is there to help the DEVs and report bugs that we may find in advance, but we have no obligation to do beta testing.

Automated testing, which is the best remedy for this kind of situation, is only really feasible with purely server-side software, or with special frameworks dedicated to your specific interfaces, such as Postman or Selenium. A unit test framework exists for Unity, but requires some knowledge of .NET that I am not sure whether the SW developers have. This would help catch some of the bugs, but not all.

However, setting up a regression test suite is a cost, particularly for a user interface. Procedures are very easy to unit test, but not GUIs.

So the trade-off here is :

1. someone having a bumpy gaming night every couple years, and then it is fixed

vs.

2. delaying important features by months (not days, because it would take man/months to automate)

I am afraid that most PR consultants would advise that #2 is more likely to induce mass migrations of customers towards Foundry, so the choice SmiteWorks made still makes sense, however disappointed it left those who suffered the disruptions. Sometimes you have to compromise. And as I have said elsewhere, the speed at which the devs fixed the bugs was remarkable.

Yes, the process has room for improvement, but don't be so harsh with team SmiteWorks. They have probably made the right call.

HywelPhillips

November 25th, 2024, 10:32

Since major releases are available in the beta channel for sometime before deploying to live, would there be a group of folks willing to run one-shots in the test channel prior to go lives?

I am willing to do that although I cannot guarantee availability to do it for every beta release. (It just depends on my filming schedule).

Hywel

HywelPhillips

November 25th, 2024, 10:44

Without wanting to be snide and cast shade, I will say that RosenMcStern's point about feature roll-out is an important one. If you think this release of FGU was bad, wait until you've gone through a version upgrade of Foundry. It's typical for rulesets not to get updated for months, and for extensions to fall out of use completely. Even the major hosting platform for Foundry, ForgeVTT, had to hold off going to version 12 for their hosted games for six months or more after its release. Typically things only catch up on a major version release around the time the next major version release emerges.

This move-fast-and-break-things approach has meant Foundry has achieved impressive functionality in a short space of time, but at the expense of the only meaningful way of running a campaign on it that I've found is to freeze everything the moment you have it working, and not update AT ALL for the duration of the campaign.

By comparison the FGU recommendation of don't update on game night is very gentle. I might extend it to "don't update until a week or two after a release of the CORE FGU software unless you have time to test before game night, and if it seems broken, switch to the PREV channel until the devs have chased down any unexpected game-breaking bugs".

I think SW do a great job but obviously the more can be caught in advance of release day, the less time the team has to spend frantically firefighting afterwards. I hate to think how many hours Moon Wizard, Carl et al have been in the office making sure we get to play our elf games come Thursday night! :)

jharp

November 25th, 2024, 12:10

Arsilon

November 25th, 2024, 13:28

I do think a lot of how FG works has specific design intents around how it was built and how it was optimized. None of that is exposed to the end user and GM's and Players are left to figure out what the limits are on their own. Most often these things are not game breaking, just perhaps inefficient. Even if they do test it, they aren't testing the biggest baddest thing to see 'if it breaks'. They are testing the simplest and easiest to understand things to see 'if it works'. Most community testers are not really testers and aren't artificially creating things to test that will break things. These are the types of things that QA should have already rooted out -- and this is where there is likely some fair blame to be assigned to SW even with a very minimal staff. There should be very large maps with obscene amounts of LOS data, etc. that QA are using to try and break things. Perhaps they already are, perhaps they aren't. This is especially true around a release that had more than usual emphasis on maps and images. Hard to say going back to my original statement in that we don't know what the exact design and performance criteria are and what was never meant to work well vs. not.

That being said, things like memory usage and size of images have always been a known and core source of performance issues. There are stated guidelines for developers that the average user or GM never even looks at and most likely never realize they are going far beyond.

In such cases, a simple pop up - "Warning, this image exceeds optimal size standards. Performance degradation blah blah blah" " before it is allowed to be converted into a map object would probably avoid 60% of the issues. (You might even consider adding it before it is actually opened -- but that might get annoying).

ps In fairness, I appreciate that SW doesn't put hard limits in what we can even try to load. But I'm also computer savvy enough to know where I will likely be creating problems for myself when I do. And know how to undo it when that happens. This isn't likely true for a significant portion of the user/player base.

fantasylord

November 25th, 2024, 13:34

I guess this is good to know. I've only ever used the platform as a player, not a GM. Trouble with our GM is he is barely computer literate so all the switching channels and "don't" update is something we'll have to work with him on. He see's a shinny "Upgrade" button and he hammers it thinking he has to update, and we all have to follow along. We play ever Sunday night, so hopefully the bugs are worked out this week. If it helps, we are playing DND 5e, the map currently is pretty large. There were 7 of us on last night, (including GM). Game would crash. Sometimes you could not move open items. You could close them and reopen them, but not move them. The freezing was the indicator that a crash was coming. Compared to the previous two platforms, we love this one so I will retract my Roll20 comment. This platform has served us countless hours of entertainment.

Trenloe

November 25th, 2024, 13:55

-Do releases Monday morning at the start of everyone's work day at Smiteworks
For those who aren't aware, release day is Tuesday.

alloowishus

November 25th, 2024, 14:27

HoratioDrank

November 25th, 2024, 14:42

People who don't bother to get in TEST before it goes LIVE and leave the burden to those of us that do? When we all know SW as a company has 3 software engineers to manage all this?

THEY REAP THE REWARDS OF THEIR OWN APATHY!

You got what you deserved for leaving TEST to rot for a month while you did nothing. I have no pity for those complaining now. My stuff works - and I see nothing major in my simple campaigns because I was there in the trenches while the rest you sods loafed waiting for the hammer to fall.

No pity here. At all.

IMHO

If your implication is that the only people who deserve a useable product are those willing to do free work for a for-profit company, I emphatically disagree in the strongest terms. It’s not my responsibility to test their product. But it is their responsibility to deliver useable, low-as-possible-bugged releases.

alloowishus

November 25th, 2024, 14:42

Tonight's game was a wash. Stupid dev's should have to use this stuff all day before they release the updates. Failed testing or no testing. Destroyed our session tonight. Starting to smell like Roll20 stuff.

The most frustrating thing is that smiteworks seems to be adding a bunch of stuff that most people are not asking for. I mean 2.5D maps? Does anyone actually use that? THe amount of work it takes to setup a battle map is tremendous, and now I have to worry about setting it up for 3D mode? Stuff like memory leaks is something that any small amount of testing would have uncovered I would have thought. My guess is that the developers are testing their own code. This is NEVER a good idea, speaking from experience. Smiteworks needs a dedicated team of testers, and no code should go out without an extensive list of testing being done, perhaps automated. If this is not possible, then they need to really curb back their major releases. I was much happier with the functionality of FG 6 months ago, the only thing I liked was adding the ping functionality (although now the space bar click use does not seem to work any more).

The main problem I see currently is the memory issue, I just added another 16 GB to my desktop but it still will consume around 10-16 gigs of memory just for FG. This seems crazy to me, especially when my maps are only a few megs at most. Even with 500 KB maps it will take 8 or 9 gigs of memory. Is this all for the 3D stuff? I would much prefer having the option of not using that if it means less memory useage.

Having said that, overall I still think it an excellent platform when it works properly, I really do say this from the perspective of a customer and a fan, but a frustrated fan.

HywelPhillips

November 25th, 2024, 15:13

I do have an ongoing subscription:

I stand corrected. I would feel more annoyed if I was paying monthly too.

In fairness, one of the things they added this release was something that has consistently been highly requested - text on maps/images.

I also agree that the scope of the memory leaks and asset window problems is something that I'm surprised wasn't caught earlier given that doing some basic session prep was enough to repeatedly hang the program for me on a top-spec Mac Studio. It does suggest an improvement to the testing regime should be strongly considered.

I' m also not sure how useful 2.5D will prove, I've turned it on as a player a few times just out of curiosity. But as the GM I can confidently say that I am only planning to use it when running commercial products where it is all set up for me with walls, LOS blocking assets and all NPC portraits ready-done. The overhead in prep is too much for me to handle. Maybe once in a while for a big boss battle, I can see fighting a big bad dragon in its lair could be spectacular - IF someone has done all the hard work of setting that all up for me.

Cheers, Hywel

ddavison

November 25th, 2024, 15:32

I want to chime in here to let you know that we hear you and we apologize for the mistakes of this rollout. We attempted to improve this release with a number of new actions and we did not execute well on those. We will be working internally on improving this for future releases.

1. New features are always being added. Not every new feature will interest every user, but it is important for us to continually expand our feature set to remain competitive.
2. We have a small dev team and this means devs do test their own code
3. New features are often paired with refactoring or upgrades to newer versions of Unity to keep code maintainable and organized
4. We added regression tests of more than 200 test items in the image system and expanded testing to the rest of our team for this release.
5. There were 5 rounds of regression testing
6. As a small company, we only really have a single machine per employee and these are often higher specs than our minimum specs. A small subset of employees have multiple systems with different OS's we support, or digital versions of these for testing
7. Our system features more than 3300 DLC with hundreds of unique assets each and we also allow homebrew which greatly expands our permutations for testing. Our testing mostly focuses on official DLC
8. Each test has to be repeated for every type of asset type we support (webp (static and animated separately), webm (static and animated separately), jpg, PNG)
9. Unrelated features get tied to each other because feature sets are difficult to silo in new builds

We are still reviewing why this release failed to meet our expectations for stability, but there are a few key takeaways we have identified.

1. Refactoring seemed to lead to the biggest issues more than "new features" did. It also required us to expand our regression testing to cover way more items and significantly increased our test case load.
2. Our regression test list was incomplete and did not cover enough cases. It is being expanded with every reported bug so those should at least be covered in future regression test plans
3. Expanding testing to all staff only resulted in a few staff members actually doing testing. Everyone else was busy with other tasks and we need to carve out time specifically and assign testing more directly.
4. We need more time between test rounds. This is difficult to do with hard deadlines. We moved the image release stuff back a month due to this but we should have probably bumped it back another month.
5. We need more verification that reported bugs in the regression test are fully resolved in new builds
6. We need to pay attention to system resource usage that might lead to a crash on end-user machines, even if it does not lead to a crash on our systems

Customer testing in the TEST channel will definitely help us with this, but we are looking internally at how we can do this better ourselves. Unfortunately, we are not in a position to hire more staff.

On the positive side, we did have a number of our staff who went above and beyond to help make the release as good as it could be. Our devs were very responsive at trying to address reported issues in a timely manner. Our support staff and moderators did a great job responding to customers and funneling those reports to our dev team for quick resolution. Hopefully we can get this process down so it is smoother in the future.

Artrem

November 25th, 2024, 15:57

Thank you Doug for the message and comments.

Honestly, I was wondering why we hadn't heard from you yet. I am glad you commented today. Sometimes just updated information and acknowledgment of what went wrong and future plans going forward gives reassurance.

I know it's a small company, but with the devs so busy working the issues, it's nice to have a point person summarizing for everyone what's happening and where we are headed. Especially when there is a significant number users having issues with the product (clearly it wasn't everyone). Your post has given me some solace about fantasy grounds; which I clearly love. I assume others do as well, otherwise, we all wouldn't be so passionate about it on the forums and in discord when it's not working well.

Again, I may not agree with all the planned features and yes, I worry when these planned features disrupt the program's established functions. Temporary setbacks will happen, as long as they are temporary and do not degrade things that already work well.

Yes, your dev team has been amazing as always, tirelessly working the issues the past six days. I know I greatly appreciate the work that allows myself and my group to continue our adventures...

Artrem
(less salty today)

alloowishus

November 25th, 2024, 17:48

Thanks for the apology, here are some suggestions from my experiences.

1. Have a dedicated dependable test team that you pay. This is not open source software, people pay for it, so I would expect it to be run like a professional business, regardless of the number of developers.
2. Create suitable environments for testing, this means Dev, Test/QA and Production (if you want to be really adventurous, seperate QA and UAT as well)
3. I understand that you need to keep the software relevant, but the number of users you are going to impress is much lower than the ones you are going to piss off if you don't ensure that you aren't breaking existing functionality.
4. If possible, introduce the ability to select which major release version the users can use. I would personally love this, since I like to spend most of my time improving my ruleset mods and campaigns rather than suffering through releases which crash because of features I don't even use.

Thanks very much.

LordEntrails

November 25th, 2024, 19:24

Thanks Doug. Appreciate the insight and let us know if we can help.

The most frustrating thing is that smiteworks seems to be adding a bunch of stuff that most people are not asking for. I mean 2.5D maps? Does anyone actually use that? THe amount of work it takes to setup a battle map is tremendous, and now I have to worry about setting it up for 3D mode?
I understand the sentiment, but I think it's an observation based on a very limited viewpoint (which is ok). As I trty to browse the wider aspects of the VTT community, I find most discussions about which VTT to chose is not about stability. It's about "flash". Very rarely does anyone complain about Foundry's stability and conflict issues. In fact, it seems like most people simple ignore it as being part of the accepted "cost of doing business". What tends to drive people to chose a VTT are: A) "Flash" as I call it. Animation and other visual immersion tools like 3D or first person view. B) Cost. Many people think software or SaaS should be free for their hobbies. c) Simplicity or more often perceived simplicity.

Very few people ever care when I talk about; platform stability, company ethics, or longevity (both past and future potential). Most of the market potential comes from the under 30 group. And they don't care about what might be in a year or ten. They want cheap, easy, and now. And something with an impressive visual appeal. They know FPS and 3D have been around for decades, so they see no reason their VTT shouldn't have it too.

2. Create suitable environments for testing, this means Dev, Test/QA and Production (if you want to be really adventurous, seperate QA and UAT as well)
These already exist, evidence of such can be seen in the publicly facing Channels on your FG application settings. Dev, Test, Live, & Prev.

3. I understand that you need to keep the software relevant, but the number of users you are going to impress is much lower than the ones you are going to piss off if you don't ensure that you aren't breaking existing functionality.

Upsetting current users is always bad. And a current user has more value than a potential new customer. But FG has well less than 475k users (based on forum users which is required to use FGU, but does not account for free only or inactive accounts). We also know the number of potential customers for D&D alone is in the neighborhood of 50 million. That's 100 to 1, easily. SmiteWorks can't ignore potential new customers. (Just as they can't and are not ignoring current customers).

4. If possible, introduce the ability to select which major release version the users can use. I would personally love this, since I like to spend most of my time improving my ruleset mods and campaigns rather than suffering through releases which crash because of features I don't even use.
This is technically difficult and probably not feasible in the long run.
A simple example, the 2024 DMG for D&D. Requires new capabilities (locations). Therefore you would leave behind customers want to run an older version as you certainly would not be able to maintain multiple versions.

I work for a company that uses a major enterprise system. The users of this software from a $23 billion dollar company have for decades asked for the same thing you are. We pay over a $1mil in license costs per year. This software has the same amount of user facing version control as we have in FG. Each user can chose to update to the new version to get the new capabilities, or keep using the old version until it is no longer supported or functions.

I think that for the most part his is a great discussion. We all know this release was bad. Doug has admitted it. But we've also seen that as has always been the case, SW devs step up and work diligently to resolve issues. And, as Doug has shown, SW has learned from this and is already implementing lessons learned into their processes to reduce the chance of this happening again.

Yes it's likely it will happen again. But hopefully it will be less likely and less severe next time. And hopefully SW will respond like they have this time and the community will also be as diligent to report issues and maintain a useful attitude.

alloowishus

November 25th, 2024, 22:38

Thanks Doug. Appreciate the insight and let us know if we can help.

I understand the sentiment, but I think it's an observation based on a very limited viewpoint (which is ok). As I trty to browse the wider aspects of the VTT community, I find most discussions about which VTT to chose is not about stability. It's about "flash". Very rarely does anyone complain about Foundry's stability and conflict issues. In fact, it seems like most people simple ignore it as being part of the accepted "cost of doing business". What tends to drive people to chose a VTT are: A) "Flash" as I call it. Animation and other visual immersion tools like 3D or first person view. B) Cost. Many people think software or SaaS should be free for their hobbies. c) Simplicity or more often perceived simplicity.

Very few people ever care when I talk about; platform stability, company ethics, or longevity (both past and future potential). Most of the market potential comes from the under 30 group. And they don't care about what might be in a year or ten. They want cheap, easy, and now. And something with an impressive visual appeal. They know FPS and 3D have been around for decades, so they see no reason their VTT shouldn't have it too.

Are you sure about that? Maybe the young ones do, but do you have statistics on your user base? Based on my campaigns, there most old timers like myself trying to relive some of the glory days. I am guessing THAT is your base, if people are looking for flash, they would go to Baldur's Gate or all the other fancy real time multi player RPGs. Everyone in my campaigns are not looking to play Baldur's Gate through a VTT, they are looking for an old school RPG experience (and if they were, I would tell them to go player Barlur's Gate instead!). Everyone in my campaign also doesn't seem to care about the 2.5D/3D features. I realise that is anecdotal, but did you see the post from the guy who was running his first campaign ever the night that of the release? What do you think his impression of FG will be? What will he/she tell their friends? IMHO, platform stability is HUGE, and is very important among my players, most do not care so much about flash, I mean FG has a ton of useful functionality already. I would love to have the ability to choose my release version when loading a campaign, you could always put in a message saying that is not supported.

My guess is that the smiteworks dev team are of the "move fast and break things" school, and that is unfortunate. I totally understand the appeal of working on cool new stuff and getting it out as quickly as possible, I have fallen into that trap many times. It takes discipline and patience to wait until something is 100% solid before releasing.

I would love to see some polls out there, I could be totally wrong about all of this. But based on the shear number of posts, I am guess I am not alone (and remember for every forum post there are probably a hundred that don't post).

LordEntrails

November 25th, 2024, 23:58

No I don't have much in the way of statistics except the two number I mentioned. You can see how many users have forum accounts here. Forum accounts are required to join a FGU game. So we know the number of users is less than 473,008. WotC claims that 50 million people have played D&D, source Google. Simple math gives you the 100:1 ratio. Yes not all D&D players are VTT players, at least not yet. Yes we know that number does not indicate current players. But we also know it does not account for the other part of the community that has never played D&D. 100:1 is an order of magnitude number, nothing more.

My point was exactly that your experience is a group looking for VTTs for the old school experience (I'm in the same demographic). But it would simply be a bad business model to focus on delivering a product that only appeals to a shrinking demographic (as "we" die or leave gaming). Given the huge growth in RPG play, again, WotC and research books from the likes of Jon Peterson and David Ewalt as well as regular industry reports from groups like Newzoo and (I can't remember the big one!) show that RPG market is growing. And the VTT market along with it (a fraction of the RPG market, but still growing).

Sure, I have no "proof" that the market growth is or is not coming from old men like us. But really, is that where you think the growth is? I don't. The growth is coming from NEW YOUNG players, and there is plenty of evidenced on non-forum social media platforms like X, Blusky, TikTok, Insta, Discord, etc. Platforms dominated by youth, again, not our demo. But posts and surveys on forums like here RPGNet, and ENWorld and such are not going to do anything but continue to mine the middle age demographic.

Yes I saw the post from the first time DM. But did you know he's been a player in FG games for many months or years? So he already knows what he experienced was not "normal". And yes its unfortunate that they had the experience that they did. But really, 100%? Nothing is 100%, ever. It's not even a goal, nor should it be. The cost, time and negative impacts as you approach 100% are exponential. If it took 8 months of development for this last release. Had SmiteWorks tried for 100% it would have taken them 8 years or something equally ridiculous. Since you don't want any new functionality, I'm sure you would be ok with that. But it would put SmiteWorks out of business.

Yes SmiteWorks needs to do better than they did on this release. Doug just said so. And that they have taken steps to improve their process. Isn't that what we want?

If you want a comparison to another niche software application, take a look at Campaign Cartographer (CC3+) from ProFantasy. Great company. Great product. But they have signed their own death warrant and they know it. Their program is based on a 32 bit kernel. They made the decision years ago that they did not have the resources to rebuild on a 64 bit kernel. Every year they fall further and further behind the curve as new mapping program like Dungeon Alchemist, Dungeon Draft, Dungeon Painter, etc eat in more and more on their market. CC used to be the ONLY dedicated RPG map making software. Now it's the OG with one foot in the grave, lung cancer and a wheelchair. It can still make some of the best maps out there, but soon the resolution of its output, the limitations in file and asset size will become something like 8-bit games. Only for those who want a retro look and a retro interface that uses a paradigm not seen elsewhere in 20 years.

I don't want to see FG die a similar death. One reason I chose FG is in hopes that the content I create for my campaigns can live on in the next generation. It can't do that if the platform is dead.

Laerun

November 26th, 2024, 01:20

Basic Analysis of Platform Challenges and possible suggestions for end user and CSR experience improvements.

Public Testing Limitations (what exactly are we testing with or against, it's too ambiguous)

Often the content and competence are missing in the users for proper engagement in public testing. The majority of the feedback concerns performance flaws i.e., without defining the reasons, that could be caused by Fantasy Grounds itself, by user errors, by competing extensions, or by the personal hardware and network setups.

Rising hardware constraints can also constrain older computer users from being able to fully test the platform.

Suggestions:

1. Develop a free test module in CoreRPG with extensive maps, sophisticated Line of Sight (LoS), lighting, and free assets. Include a reference manual with instructions, surveys, and troubleshooting resources.

2. Include this bench mark module in the demo content or release it as an independent test module.

3. Provide small incentives (eg, store points or badges) for test/ survey participation.

4. Provide clear guidance and assurances that live data will remain unaffected during beta testing, with simple instructions for creating backups.

Platform Updates and Content Separation:

For users the platform must be updated in order to have access to purchased content, frustrating them if updates bring bugs or interrupt ongoing campaigns.

Patches often cause third party extensions to stop working, and they force users out of the old stable versions.

Providing support for multiple platform versions, extensions, and rule sets is a big resource and planning task.

Suggestions:

1. Uncouple content updates (e.g., maps, tokens, DLC) from platform patches so users are able to buy and play new content even without an immediate platform patch.

2. Develop an extension manager to track installed extensions, mark buggy versions, and provide users with the option to roll back updates.

3. This keeps the platform stable for accuracy and has a dedicated beta channel for evolving features and upgrades.

4. Communication and Education:

Platform limitations, such as hardware requirements, map size constraints, and token limits, are not clearly communicated to users, leading to misunderstandings and frustration.

Critical information is distributed among forums, Discord server, Atlassian documentation, and the like in a manner that is hard for users to access reliable advice.

Suggestions:

1. Consolidate all support resources into a centralized hub with standardized FAQs, troubleshooting guides, and optimization tips.

2. Provide clear, concise guidelines about system requirements, map size limits, and best practices for campaign management.

3. Use in-app warnings to alert users about resource-heavy configurations that may impact performance.

4. Keep users informed on changes to the platform, current issues, and planned repairs via newsletters, forums and Discord.

5. Common Customer Service Challenges

Top 5 Reported Issues:

1. Network data that is blocked due to port forwarding, firewalls, or ISP limit.

2. Issues of performance, such as large maps, large number of tokens, and high-resolution assets.

3. Extension breakage caused by platform updates, disrupting user workflows.

4. Lack of accessible information due to decentralized support resources.

5. Billing problems and purchase problems (e.g., payment errors, geographic limitations) account.

Suggestions:

1. Develop automation tools for the detection and resolution of networking problems and a performance analyzer for the discovery and recommendation of best practices for campaigns.

2. Partner with extension developers to certify widely used extensions to maintain platform update compatibility and to lead them through beta testing processes.

3. Reduce account and purchase management complexities by tackling, in general, payment issues and automating, in some way, account recovery management.

4. Addressing Community Decentralization

The proliferation of community-based support (e.g., FG Academy and third-party Discord servers) has fractured the support landscape, making it potentially confusing which place of refuge to look to.

Although these efforts fill some gaps, they weaken the prominence of official materials as well as reduce consistency in the messaging.

Suggestions:

1. Collaborate with community leaders on embedding third-party activities into formal resources, and clearly link them to formal channels.

2. Recognize the importance of community-driven efforts, while actively seeking opportunities to team up, on a way that can be supported by SmiteWorks.

The above is related to a recently proposed suggestion, something that I have personally and specifically requested.

Ulric

November 26th, 2024, 03:12

SW, Please buy a M4 Mac MINI you currently only test using Moon Wizards Intel Mac. I have reported bugs that you were unaware of because you only had the Intel processor.

Lo Zeno

November 26th, 2024, 12:58

SW didn't force anyone to move to 4.6.0. Anyone can wait until others have experienced/reported the bugs and then move to it when it is more stable. As a rule always wait a month before moving to a .0 release.

That's a very disingenous take.
Since FGU has a "all or nothing" update process, if a player wants the fixes for the (for example) D&D Player's Handbook and Dungeon Master Guide (or any other module/dlc that they purchased and that receives bugfixes and updates) that have been released on the days immediately before and after the release 4.6.0, they are indeed forced to move to 4.6.0.
A whole month waiting before updating means plenty of bugs, typos, missing class features, maps with issues, etc, in purchased content that a user won't see resolved for the same whole month unless they also update the base program. A user also can't purchase any new content for the whole month, because in order to install that content they have to (guess what) update the base program, too.

Unless someone makes use of UpDEnhanced, which is a paid extension, not endorsed by SW.

HywelPhillips

November 26th, 2024, 13:40

SW, Please buy a M4 Mac MINI you currently only test using Moon Wizards Intel Mac. I have reported bugs that you were unaware of because you only had the Intel processor.

This thread is getting a bit overwhelmed with suggestions, but I'd certainly second this one. The intel macs are getting more and more outdated; M-macs launched four years ago and I doubt many consumers have purchased an Intel mac since then. Apple finally discontinued the last Intel MacPro in 2023.

The bottom-end M4 Mac mini easily satisfies the recommended instalment requirements, UK price £599.

Cheers, Hywel

jharp

November 26th, 2024, 15:40

UpDEnhanced is 33% off atm. So you can wait that month and still get updates you want.

Jason

RosenMcStern

November 26th, 2024, 18:46

I have seen some good suggestions in this thread. However, there are also some that are a bit off target.

I don't like showing credentials, but this is becoming very technical. We are discussing subjects related to industry standard practices, and experience and qualifications matter here. I have spent the last few years being in charge of releases for multi-million dollar software systems, with developer teams ten times the size of SmiteWorks. I have been in charge of an infrastructure that included, among DEV, QA, UAT and PROD a total of seventeen environments, with thirty WebLogic nodes running under load balance. In terms of computational power, it is the equivalent of the whole infrastructure of Roll20, which is the VTT market leader. And I have worked on both client/server software in C and C++ and web application systems in Java or PHP. So I know what I am talking about.

2. Create suitable environments for testing, this means Dev, Test/QA and Production (if you want to be really adventurous, seperate QA and UAT as well)

This is industry standard practice for server/based applications like Roll20 and Foundry. For those systems, it would be sound, tested advice. But for Fantasy Grounds???

The PROD environment is simply not managed by SmiteWorks. It is our desktop computers. What do you mean by create a suitable production environment? Also, UAT makes sense if there are one or more corporate entities that must give their formal green light to a release. But for a consumer product?

4. If possible, introduce the ability to select which major release version the users can use. I would personally love this, since I like to spend most of my time improving my ruleset mods and campaigns rather than suffering through releases which crash because of features I don't even use.

As already stated, this is not the current industry standard practice. Most software vendors nowadays provide free support only for the current release and the previous one at most. All previous releases must be considered unsupported, except for corporate entities that pay multi-million dollar maintenance bills. As Fantasy Grounds has no corporate users that can afford to pay big maintenance bills, we consumer users must come to terms with the fact that updating to at least "latest release -1" is mandatory, and updating to the latest is recommended, even if it may cause the occasional crash. The alternative would be worse.

My guess is that the smiteworks dev team are of the "move fast and break things" school, and that is unfortunate. I totally understand the appeal of working on cool new stuff and getting it out as quickly as possible, I have fallen into that trap many times. It takes discipline and patience to wait until something is 100% solid before releasing.

The "move fast" school is better called the Agile movement, and is a solid, tested and widely accepted approach to developing software systems. It has been around for more than 20 years. It advocates releasing value to the end users as frequently as possible, making liberal use of the incremental approach and taking advantage of user feedback as often as possible. And it takes discipline to adopt it, even if it may seem chaos to those who don't grok it.

Sure, there are safeguard mechanisms developed over time to minimise the disadvantages of tight release cycles, for instance the DevSecOps automated build/test pipelines that detect bugs before they reach end users. But they are not 100% efficient, and not so adapted to client/server systems. And, most important of all, it seems that SmiteWorks is making an effort to adopt them.

So, yes, the last release had bugs, and the process has room for improvement. But the SmiteWorks teams is going dowwn the correct route, believe me.