Sunday, July 2, 2017

Azure Elastic Pool Databases and ARM Templates

Recently, I've been working with dynamically creating elastic pool databases based on customer need/demand.  The goal of this post is to discuss some of the aspects of this process.

What are elastic pools?

Essentially, elastic pool databases are an effective way of managing many databases that have varying usage/demand profiles.  Instead of thick-provisioning compute capacity on a per database perspective, you can create a "pool" of resources and then deploy many databases on top of this pool.  For more information, here is a link to the documentation.

What is the architecture of elastic pools?

Logically, Azure has the concept of a "sql server" that can then host many elastic pools and/or regular sql databases.  At the server level, authentication/backup/audit are configured.  Each logical resource deployed on that then inherit those properties.  At the pool level, performance targets are set (in eDTUs), diagnostics can be configured, and IAM policies can be set.  If you are deploying a pool, you can then deploy several databases on top of this.  These databases are the logical contains for data, handling authentication/access concerns at that level.

Elastic pools suffer from the noisy neighbours problem, and due care in the architecture should be considered.

Okay, lets build the server

For my deployment, I decided to create a separate sql server ARM template to handle that concern.  I am big into powershell orchestration, and my scripts are already using this as the deployment technology.  As such, it is easy for me to separate these concerns while keeping a clean process for the end user.

Here is the ARM template that I have used.

    {
      "comments": "The sql server",
      "type": "Microsoft.Sql/servers",
      "name": "[parameters('sqlServerName')]",
      "location": "[resourceGroup().location]",
      "apiVersion": "2015-05-01-preview",
      "dependsOn": [],
      "tags": {
        "displayName": "shared sql"
      },
      "properties": {
        "administratorLogin": "[parameters('sqlServerLogin')]",
        "administratorLoginPassword": "[parameters('sqlServerPassword')]",
        "version": "12.0"
      },
      "identity": {
        "type": "SystemAssigned"
      },
      "resources": [
        {
          "name": "AllowAllWindowsAzureIps",
          "type": "firewallRules",
          "location": "[resourceGroup().location]",
          "apiVersion": "2014-04-01",
          "dependsOn": [
            "[parameters('sqlServerName')]"
          ],
          "properties": {
            "startIpAddress": "0.0.0.0",
            "endIpAddress": "0.0.0.0"
          }
        }
      ]
    }

At the server level, there are essentially two things to configure.  The first is the default username/password, and the second is the sql firewall.  For the password, I chose to pass in the password via a securestring parameter.  As this script is designed to dynamically build servers and pools, I need somewhere to handle the password concern without the need for human intervention.  Here is a powershell snip that shows the code to integrate with Azure KeyVault.


function GetTempPassword([int]$length) {
 $ascii=$NULL;For ($a=33;$a –le 126;$a++) {$ascii+=,[char][byte]$a }
 for ($loop=1; $loop –le $length; $loop++) {
            $TempPassword+=($ascii | GET-RANDOM)
    }
 return $TempPassword
}

function StorePasswordIn($keyVaultName,$secretName,$password){
    "Converting to secure string"
    $securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force
    "Setting in keyvault $secretName"
    Set-AzureKeyVaultSecret -VaultName $keyVaultName `
                            -Name $secretName `
                            -SecretValue $securePassword 
}

function GetStoredPassword($keyVaultName,$secretName){
 $secrets = Get-AzureKeyVaultSecret -VaultName $keyVaultName
 if ($secrets.Name -notcontains $secretName){
  $unsecuredPassword = (GetTempPassword -length 30)
  StorePasswordIn -keyVaultName $keyVaultName -secretName $secretName -password $unsecuredPassword
 }
 return (Get-AzureKeyVaultSecret -VaultName $keyVaultName -Name $secretName).SecretValue
}

Basically, I needed to handle the case where the script was being run against an already created database.  This is important as I want to be able to incrementally add elastic pools to my existing server as required.

I feel like a better approach to passing the password via securestring parameter is to use the keyvault integration in ARM.  See this link.  A planned upgrade for sure!

Now for the elastic pools

Here is my template for an elastic pool:

    {
      "comments": "Resource for each elastic pool",
      "name": "[concat(parameters('sqlServerName'),'/',parameters('elasticPoolNames')[copyIndex()])]",
      "type": "Microsoft.Sql/servers/elasticPools",
      "location": "[resourceGroup().location]",
      "apiVersion": "2014-04-01",
      "tags": {
        "displayName": "elastic pools"
      },
      "properties": {
        "edition": "[parameters('editions')[copyIndex()]]",
        "dtu": "[parameters('dtus')[copyIndex()]]"
      },
      "copy": {
        "name": "elasticPoolCopy",
        "count": "[length(parameters('elasticPoolNames'))]"
      },
      "dependsOn": [
      ]
    },

The interesting part here is being able to dynamically add pools.  Essentially, I pass in 3 arrays into this script that contain the required information.


    "elasticPoolNames": {
      "type": "array",
      "metadata": {
        "description": "The names of the pools to create"
      }
    },
    "editions": {
      "type": "array",
      "metadata": {
        "description": "The edition of the pools"
      }
    },
    "dtus": {
      "type": "array",
      "metadata": {
        "description": "The DTUs for the pools"
      }
    },

This way, my script can be used to either (a) scale existing pools as required or (b) create new pools as requirements dictate.

Add some alerts for good measure

Essentially, we are creating our own "shared infrastructure" for the various applications/clients that are running on the given pool.  Decisions need to be made at a certain point as to if the solution should be vertically or horizontally scaled.  In order to support reactive scaling, I've created alerts on the following metrics:
1) dtu_consumption_percentage
2) storage_percentage
3) sessions_percentage

The first two in the list are directly linked to the size of the elastic pool and could trigger a scale vertical/horizontal decision.  Sessions, on the other hand, are fixed for all sizes.  Reaching this limit could trigger a horizontal scale decision.

Here is the ARM template for that:

   {
      "comments": "Adding Session Alerts",
      "name": "[concat(parameters('elasticPoolNames')[copyIndex('elasticPool')],'_','sessions_percent')]",
      "type": "Microsoft.Insights/alertrules",
      "location": "[resourceGroup().location]",
      "apiVersion": "2016-03-01",
      "tags": {
        "displayName": "session alert"
      },
      "properties": {
        "name": "[concat(parameters('elasticPoolNames')[copyIndex('elasticPool')],'_','sessions_percent')]",
        "description": "an alert rule",
        "isEnabled": true,
        "condition": {
          "odata.type": "Microsoft.Azure.Management.Insights.Models.ThresholdRuleCondition",
          "dataSource": {
            "odata.type": "Microsoft.Azure.Management.Insights.Models.RuleMetricDataSource",
            "resourceUri": "[concat(resourceId('Microsoft.Sql/servers', parameters('sqlServerName')), '/elasticPools/',parameters('elasticPoolNames')[copyIndex()])]",
            "metricName": "sessions_percent"
          },
          "threshold": 90,
          "windowSize": "PT10M"
        }
      },
      "dependsOn": [
        "[concat(resourceId('Microsoft.Sql/servers', parameters('sqlServerName')), '/elasticPools/',parameters('elasticPoolNames')[copyIndex()])]"
      ],
      "copy": {
        "name": "elasticPool",
        "count": "[length(parameters('elasticPoolNames'))]"
      }
    },
    {
      "comments": "Adding DTU Alerts",
      "name": "[concat(parameters('elasticPoolNames')[copyIndex('elasticPool')],'_','dtu_consumption_percent')]",
      "type": "Microsoft.Insights/alertrules",
      "location": "[resourceGroup().location]",
      "apiVersion": "2016-03-01",
      "tags": {
        "displayName": "dtu alert"
      },
      "properties": {
        "name": "[concat(parameters('elasticPoolNames')[copyIndex('elasticPool')],'_','dtu_consumption_percent')]",
        "description": "an alert rule",
        "isEnabled": true,
        "condition": {
          "odata.type": "Microsoft.Azure.Management.Insights.Models.ThresholdRuleCondition",
          "dataSource": {
            "odata.type": "Microsoft.Azure.Management.Insights.Models.RuleMetricDataSource",
            "resourceUri": "[concat(resourceId('Microsoft.Sql/servers', parameters('sqlServerName')), '/elasticPools/',parameters('elasticPoolNames')[copyIndex()])]",
            "metricName": "dtu_consumption_percent"
          },
          "threshold": 90,
          "windowSize": "PT10M"
        }
      },
      "dependsOn": [
        "[concat(resourceId('Microsoft.Sql/servers', parameters('sqlServerName')), '/elasticPools/',parameters('elasticPoolNames')[copyIndex()])]"
      ],
      "copy": {
        "name": "elasticPool",
        "count": "[length(parameters('elasticPoolNames'))]"
      }
    },
    {
      "comments": "Adding Storage Alerts",
      "name": "[concat(parameters('elasticPoolNames')[copyIndex('elasticPool')],'_','storage_percent')]",
      "type": "Microsoft.Insights/alertrules",
      "location": "[resourceGroup().location]",
      "apiVersion": "2016-03-01",
      "tags": {
        "displayName": "storage alert"
      },
      "properties": {
        "name": "[concat(parameters('elasticPoolNames')[copyIndex('elasticPool')],'_','storage_percent')]",
        "description": "an alert rule",
        "isEnabled": true,
        "condition": {
          "odata.type": "Microsoft.Azure.Management.Insights.Models.ThresholdRuleCondition",
          "dataSource": {
            "odata.type": "Microsoft.Azure.Management.Insights.Models.RuleMetricDataSource",
            "resourceUri": "[concat(resourceId('Microsoft.Sql/servers', parameters('sqlServerName')), '/elasticPools/',parameters('elasticPoolNames')[copyIndex()])]",
            "metricName": "storage_percent"
          },
          "threshold": 90,
          "windowSize": "PT10M"
        }
      },
      "dependsOn": [
        "[concat(resourceId('Microsoft.Sql/servers', parameters('sqlServerName')), '/elasticPools/',parameters('elasticPoolNames')[copyIndex()])]"
      ],
      "copy": {
        "name": "elasticPool",
        "count": "[length(parameters('elasticPoolNames'))]"
      }
    }



In conclusion, we talked about some considerations for elastic pool architecture and shared some ARM templates for the dynamic creation of both sql servers and elastic pools. Lastly, we talked a bit about alerts that could be used to support vertical/horizontal scale decisions.

85 comments:

  1. Thanks for sharing the post.. parents are worlds best person in each lives of individual..they need or must succeed to sustain needs of the family. swimming pool service technician near me

    ReplyDelete
  2. You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. swimming pool services near me

    ReplyDelete
  3. Great write-up, I am a big believer in commenting on blogs to inform the blog writers know that they’ve added something worthwhile to the world wide web!.. 바카라사이트

    ReplyDelete
  4. Manual pool cleaning is tedious and arduous which is the reason many pool proprietors put resources into a programmed pool more clean. First names only

    ReplyDelete
  5. Be that as it may, today there are as yet numerous undertakings who need us to give ensures on accomplishing page one positioning for watchwords. blog comments service in 1$

    ReplyDelete
  6. I’m going to read this. I’ll be sure to come back. thanks for sharing. and also This article gives the light in which we can observe the reality. this is very nice one and gives indepth information. thanks for this nice article... custom pool builder spicewood tx

    ReplyDelete
  7. I think this is one of the most significant info for me. And i’m glad reading your article. But want to remark on some general things, The web site style is ideal, the articles is really great : D. Good job, cheers laptop mockup

    ReplyDelete
  8. So lot to occur over your amazing blog. Your blog procures me a fantastic transaction of enjoyable.. Salubrious lot beside the scene. Achtformbecken

    ReplyDelete
  9. Information on common services provided during regular weekly maintenance. Also, how to choose a quality provider and how much you should pay for weekly pool service. EasySet

    ReplyDelete
  10. Thank you for some other informative website. The place else may just I get that kind of information written in such a perfect method? I have a venture that I am simply now running on, and I’ve been at the glance out for such info. What to put under an Intex pool

    ReplyDelete
  11. Remarkable article, it is particularly useful! I quietly began in this, and I'm becoming more acquainted with it better! Delights, keep doing more and extra impressive! Ovalpools

    ReplyDelete
  12. An excellent pool for kids and adults certainly comes with great water toys. Pool

    ReplyDelete
  13. This is just the information I am finding everywhere. Thanks for your blog, I just subscribe your blog. This is a nice blog.. Aufstellbecken

    ReplyDelete
  14. For the individuals who have a family, at that point likely for the greater part of the occasions, the pool will be utilized by minimal ones.Edelstahlpool

    ReplyDelete
  15. Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up. instagram likes buy cheap

    ReplyDelete
  16. Uncommon tips and clear. This will be to a great degree supportive for me when I get a chance to start my blog. Aufstellpools

    ReplyDelete
  17. On the off chance that it is an automated spread framework, any limited quantity of standing water on the spread will slide off as you move it up.Holzpool

    ReplyDelete
  18. If your TA level is too high your water can become cloudy, the PH level will be hard to keep balanced, chlorine will not be as effective and your pool will constantly need more acid.Stahlwandpool

    ReplyDelete
  19. Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. Ovalpools

    ReplyDelete
  20. In addition, if perhaps you do not keep your pool available year round, a pool service can certainly also open and close your pool for an added cost. Schwimmbecken

    ReplyDelete
  21. Common stone likewise demonstrates to give a superior surface than stepped concrete since it isn't tricky and the shading doesn't blur. Swimmingpool

    ReplyDelete
  22. Students learn about the different Virtual Machine templates in the gallery. They get to explore configuration, management and monitoring techniques. AZ-900 courses

    ReplyDelete
  23. Pool cleaning instruments show the advancement of man. Pool cleaning devices demonstrate that man can make answers for any difficult that comes his direction. link

    ReplyDelete
  24. If you want to enjoy the best online casino game you've never experienced before, click the link you've hit now! You can experience enormous events and online casino sites with the largest members.
    안전카지노사이트
    카지노사이트
    카지노검증사이트
    바카라검증사이트
    실시간카지노

    ReplyDelete
  25. I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. temporary pool fencing perth

    ReplyDelete
  26. They are regularly ready to withstand most climate occasions, including solid, breezy breezes.Pool Fencing Gold Coast

    ReplyDelete
  27. Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. Holzpool

    ReplyDelete
  28. Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share. Achtformbecken

    ReplyDelete
  29. Thanks for sharing the post.. parents are worlds best person in each lives of individual..they need or must succeed to sustain needs of the family. Aufstellpools

    ReplyDelete
  30. This interaction can require a little as about fourteen days in the slow time of year or however much a month and a half or really throughout the spring and summer season.Holzpool

    ReplyDelete
  31. Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. Artikelmap

    ReplyDelete
  32. Hello, this weekend is good for me, since this time i am reading this enormous informative article here at my home. https://pool.net/gartenpool/

    ReplyDelete
  33. Your website is really cool and this is a great inspiring article. Edelstahlpool

    ReplyDelete
  34. everyday work. His rivals will trust that his fall will proceed in the Jerusalem town hall where he is now being investigated on genuine debasement accusations온라인카지노

    ReplyDelete
  35. Doing Pool Pump Repair properly involves knowing the ones that you can handle on your own and those that require a professional. Pool damage can be prevented with regular pool maintenance.

    ReplyDelete
  36. Really appreciate you sharing this blog article.Really thank you! Will read on スロット


    ReplyDelete
  37. Impressive! I am amazed at how well you use words to get your point across. I would be interested in reading more of your work. 에볼루션카지노

    ReplyDelete
  38. I’m excited to uncover this website. I wanted to thank you for your time due to this wonderful read!! I definitely really liked every bit of it and I have you saved as a favorite to check out new stuff in your website. 메이저놀이터

    ReplyDelete
  39. I’m impressed, I must say. Seldom do I encounter a blog that’s both equally educative and amusing, and let me tell you, you’ve hit the nail on the head. The issue is something which not enough folks are speaking intelligently about. I am very happy that I stumbled across this during my hunt for something relating to this. 안전놀이터

    ReplyDelete
  40. Thank you so much for the post you do. I like your post and all you share with us is up to date and quite informative, i would like to bookmark the page so i can come here again to read you, as you have done a wonderful job..I haven’t any word to appreciate this post.....Really i am impressed from this post....the person who create this post it was a great human..thanks for shared this with us 먹튀폴리스

    ReplyDelete
  41. ove to read it,Waiting For More new Update and I Already Read your Recent Post its Great Thank..great article. thank you for sharin..Great post. Thank You For Sharing Valuable. information. It is Very Informative article..I urge you to peruse this content it is fun portrayed .. 해외안전놀이터

    ReplyDelete
  42. What a nice blog post, and learned alot..Hi, I do believe this is a great blog. I stumbledupon it 😉 I am going to come back yet again since i have book-marked it. Money and freedom is the best way to change, may you be rich and continue to guide others..Pretty! This has been an incredibly wonderful post. Many thanks for supplying this information. 먹튀검증

    ReplyDelete
  43. Newstyle Print offers high end, low cost banner printing in the UK.. like review goals which understand the cost of passing on the amazing strong asset futile out of pocket. I truly worshiped examining your posting. Grateful to you! 슈어맨도메인

    ReplyDelete
  44. I really discovered this a rich degree of data. The thing I was looking through for.I ought to recommend you that please continue sharing such sort of data..I should thank you for the undertakings you have made in making this post. I'm confiding in an equivalent best work from you later on also. I need to thank you for this site! Appreciative for sharing. Immense domains. 먹튀썰전

    ReplyDelete
  45. On my website you'll see similar texts, write what you think..Thank you so much for the post you do. I like your post and all you share with us is up to date and quite informative, i would like to bookmark the page so i can come here again to read you, as you have done a wonderful job 안전토토사이트

    ReplyDelete
  46. I at last discovered grand post here.I will get back here. I just added your blog to my bookmark districts. thanks.Quality presents is the major on welcome the guests to visit the site page, that is the thing that this site page is giving.I see something fascinating about your website, so I saved to bookmarks. Web webpage: 먹튀검증

    ReplyDelete
  47. Quality content is the main to be a focus for the people to visit the web..page, that’s what this web site is providing.I am sure this paragraph has touched all the internet visitors,..its really really good piece of writing on building up new.Heya i’m for the primary time here. I came across this board and I find It really useful & it helped me out much. 메이저사이트

    ReplyDelete
  48. Thank you for helping people get the information they need. Great stuff as usual. Keep up the great work.. This is a fantastic website, thanks for sharing. There's no doubt i would fully rate it after i read what the idea about this article is. You did a nice 바카라사이트

    ReplyDelete
  49. I think this is one of the most significant info for me. And i am glad reading your article.But wanna remark on some general things, The web site style.is wonderful, the articles is really nice : D. Good job, cheers.I read this paragraph fully about the comparison of latest and previous technologies, 토토사이트

    ReplyDelete
  50. I am thankful for the article post. Really looking forward to read more. Will read on...Superb brief which write-up solved the problem a lot. Say thank you We searching for your data...A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. 안전놀이터

    ReplyDelete
  51. I think this is one of the most significant info for me. And i am glad reading your article.But wanna remark on some general things, The web site style.is wonderful, the articles is really nice : D. Good job, cheers.I read this paragraph fully about the comparison of latest and previous technologies, 먹튀검증사이트

    ReplyDelete
  52. A great content material as well as great layout. Your website deserves all of the positive feedback it’s been getting. I will be back soon for further quality contents..I read a article under the same title some time ago, but this articles quality is much, much better. How you do this.. 먹튀검증

    ReplyDelete
  53. This is the reason it really is greater you could important examination before creating. It will be possible to create better write-up like this. .Great post and amazing facts right here.Keep it up the wonderful work..These guys are awesome ..This is a fantastic website, thanks for sharing. There’s no doubt i would fully rate it after i read what the idea about this article is. You did a nice jo 먹튀검증

    ReplyDelete
  54. You can definitely see your expertise within the work you write. The sector hopes for more passionate writers like you who are not afraid to mention how they believe. Always go after your heart..Normally I do not read post on blogs, however I wish.to say that this write-up very compelled me to check.out and do so! Your writing style has been amazed me..Thanks, very nice article.Two foot fetish guys, one of which wanted me to smear peanut butter on my feet. 안전놀이터

    ReplyDelete
  55. Well I truly liked reading it. This article provided by you is very effective for good planning..The Kansas City Metro has a lot to present when it comes to blog job possibilities.Truly quite intriguing post. I was searching for this sort of data and delighted in perusing this one. Continue posting. Much obliged for sharing 안전놀이터

    ReplyDelete
  56. Very cool website!! Man .. Beautiful .. Superb .. I’ll bookmark this site and take the feeds also…I’m satisfied to find numerous useful information here within the article. Thanks for sharing… There is noticeably a bundle to find out about this. I assume you made sure nice factors in options also. I am all that much satisfied with the substance you have said. I needed to thank you for this extraordinary article. 우리카지노

    ReplyDelete
  57. I am hoping for the same best effort from you in the future as well. In fact, your creative writing skills have inspired me.Digital Twin California.Here you can find youngsters of perfection who make some incredible memories fulfilling your dreams. You could call this spot heaven! Our high-class Goa Escorts...Thank you for very usefull information.. 사설토토사이트

    ReplyDelete
  58. Appreciate it for this post, I am a big fan of this web site would like to go on updated...Your mode of telling everything in this paragraph is really pleasant, all can easily know it, Thanks a lot...These are in fact enormous ideas in on the topic of blogging.You have touched some fastidious factors here. Any way keep up wrinting. 토토사이트목록

    ReplyDelete
  59. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info.  This is the perfect post. It helped me a lot. If you have time, I hope you come to my site and share your opinions. Have a nice day. For this site, you will see our record, make sure to experience this information. 메이저안전놀이터

    ReplyDelete
  60. We are a group of volunteers and opening a new scheme in our community. Your website offered us with valuable info to work on. You’ve done a formidable job and our entire community will be thankful to you. 먹튀검증

    ReplyDelete
  61. Stunning! Such an astonishing and accommodating post this is. I super love it. It's so great thus amazing. I am simply stunned. I trust that you keep on doing your work like this later on moreover.  온라인카지노

    ReplyDelete
  62. Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share 먹튀검증

    ReplyDelete
  63. I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog, I will keep visiting this blog very often 먹튀사이트

    ReplyDelete
  64. Great articles and great layout. Your blog post deserves all of the positive feedback it's been getting. 온라인슬롯

    ReplyDelete
  65. This site is excellent and so is how the subject matter was explained. I also like some of the comments too. Looking forward to your next post. 토토랜드

    ReplyDelete
  66. Stunning! Such an astonishing and accommodating post this is. I super love it. It's so great thus amazing. I am simply stunned. I trust that you keep on doing your work like this later on moreover.  먹튀검증

    ReplyDelete
  67. Stunning! Such an astonishing and accommodating post this is. I super love it. It's so great thus amazing. I am simply stunned. I trust that you keep on doing your work like this later on moreover.  프리미어리그중계

    ReplyDelete
  68. Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic 토토커뮤니티

    ReplyDelete
  69. Very useful information shared in this article, nicely written! I will be reading your articles and using the informative tips. Looking forward to read such knowledgeable articles. 모바일바다이야기

    ReplyDelete
  70. This article gives the light in which we can observe the reality. This is very nice one and gives indepth information. Thanks for this nice article. 사설토토

    ReplyDelete
  71. you have done a great job. I will definitely dig it and personally recommend to my friends. I am confident they will be benefited from this. 슬롯사이트

    ReplyDelete
  72. I felt exceptionally glad while perusing this site. This was truly exceptionally enlightening site for me. I truly preferred it. This was truly a sincere post. Much obliged!.  reformas integrales zaragoza

    ReplyDelete
  73. I Went ALL IN On 메이저추천 For A HUGE Comeback! (PROFIT!)

    ReplyDelete
  74. I generally check this kind of article and I found your article which is related to my interest.Cheap Swimming Pools Genuinely it is good and instructive information. Thankful to you for sharing an article like this.

    ReplyDelete
  75. The collection of real-time sports broadcasting sites provides live overseas soccer, basketball, baseball, and volleyball broadcasts for free. You can watch real-time sports games without logging in on your PC and mobile sports broadcasting in the Internet environment. Sports live scores, sports analysis picks, live chat and fast sports broadcasting sites provide satisfactory game broadcasting videos to soccer, basketball, and Molve broadcast users. 스포츠중계 sportsnewslives

    ReplyDelete
  76. Toto Games is divided into online gaming sites Private Toto and Batman Toto. The Toto Center is defined as a haven of peace and provides a mobile casino for gambling users. The most profitable place for sports commentators using Toto and Protobetting. 토토사이트 mujuru 안전놀이터

    ReplyDelete
  77. Sports Toto is divided into Private Toto and Batman Toto, Korean online betting sites. Major Toto sites are defined as safety playgrounds and serve the mobile betting environment for sports betting users. It is the most useful website for sports analysts who use Toto and Protobetting. 토토사이트 토토사이트 모음 안전놀이터

    ReplyDelete
  78. The cost-effective side of Toto brings users of Korean sports Toto reasonable dividends and beneficial events. Important Security Games, Private Sites This and Toto recommended by the community for food and run verification are considered to be the most useful online betting sites. 토토사이트 사설토토 안전놀이터

    ReplyDelete
  79. The cost-effective Toto site provides reasonable dividends and useful events to Korean Sports Toto users. Major safety playgrounds, private Toto, and Toto sites recommended by the food and run verification community are considered the most useful websites among online betting sites. 토토사이트 안전 토토사이트 안전놀이터

    ReplyDelete
  80. I recommend the Toto site where Toto users can bet one folder and multiple folders. If you are using Sports Toto betting on a saved stadium, you can bet on plane tickets easily without having to worry about taking a run or delaying the exchange. The best betting sites in Korea are listed in the sports betting category. 토토사이트 안전 메이저놀이터 안전놀이터

    ReplyDelete
  81. This website is usually a walk-through you discover the details it suited you about this and didn’t know who need to. Glimpse here, and you’ll undoubtedly discover it.bathroom renovations

    ReplyDelete
  82. The West must prepare to continue supporting Ukraine in a war lasting for years, Nato's chief has warned.

    ReplyDelete
  83. I read a article under the same title some time ago, but this articles quality is much, much better. How you do this.. lifeguard certificate near me

    ReplyDelete